I am having issues figuring out the correct way to reformat/convert date fields in the js.
I am trying to create a report using the Kafka ODF streamer and sink connector and a js in between to handle transform, from an input EDR file.
Sample O/P report
MSISDN,DeleteReason,DeleteTime
12554564364661,3,2022-02-21T14:08:39.795Z
Expected report format
MSISDN,DeleteReason,DeleteTime
12554564364661,3,20220221140839
Note:- Want to transform the DeleteTime field to a different format "YYYYMMDDhhmiss", currently YYYY-MM-DDTHH:MM:SS.MISZ .
Transform section in the sink connector
transforms: valueTransform
transforms.valueTransform.type: "com.openet.odf.transformer.ValueTransform"
transforms.valueTransform.script: "/opt/kafka/external-configuration/sink-hookpoint/valueTransform.js"
eventTimestamp[or DeleteTime] is an incoming date field declared as a string in the schema. ex: 2022-02-21T14:08:39.795Z
Js Script
valueTransform.js: |
function onValueTransform(_schema, _value) {
var SchemaAndValue = Java.type('org.apache.kafka.connect.data.SchemaAndValue')
var Date = Java.type('java.util.Date')
var d = new Date.parse(_value.get("eventTimestamp"))
var year = d.getFullYear()
var month = ('0' + (d.getMonth() + 1)).slice(-2)
var day = ('0' + d.getDate()).slice(-2)
var hours = ('0' + d.getHours()).slice(-2)
var minutes = ('0' + d.getMinutes()).slice(-2)
var seconds = ('0' + d.getSeconds()).slice(-2)
var modified_delete_time = year + month + day + hours + minutes + seconds
_value.put("eventTimestamp",modified_delete_time)
return new SchemaAndValue(_schema, _value)
}
I've tried using Date, SimpleDateFormat etc, but none of it is working and getting different errors with each attempt.
Thanks, Forum.
I was able to use Calendar and SimpleDateFormat to achieve the desired results. Solution below
valueTransform.js: |
function onValueTransform(_schema, _value) {
var SchemaAndValue = Java.type('org.apache.kafka.connect.data.SchemaAndValue')
var Date = Java.type('java.util.Date')
var SimpleDateFormat = Java.type('java.text.SimpleDateFormat')
var Calendar = Java.type('java.util.Calendar')
var sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss")
var EventDate = sdf.parse(_value.get("eventTimestamp"))
sdf=new SimpleDateFormat("yyyyMMddHHmmss")
var cal = Calendar.getInstance()
cal.setTime(EventDate)
EventDate = cal.getTime()
var formattedDate = sdf.format(EventDate)
_value.put("eventTimestamp",formattedDate)
return new SchemaAndValue(_schema, _value)
}
There's nothing reasonable built-in. You can use Moment Timezone for Moment.js in Javascript.
var actualDateTime = moment("2022-02-21T14:08:39.795Z");
var f = "YYYYMMDDhhmmss";
console.log(actualDateTime.format(f)) // By passing format into the format function to get it done
In most of the cases, dealing with Dates may be a tricky process.
I would consider choosing a Dates library, for instance https://www.npmjs.com/package/luxon.
My first choise would be Moment.js, but the project is deprecated.
Using Luxon, you can rely on parsing and formatting utilities, something like that:
DateTime.fromISO('2022-02-23T08:30:00').toFormat('yyyyMMddhhmmss')