Using Listeners is one of the easiest way to integrate with Herogi. You can easily configure a listener with an event mapping and then run it to send events in real-time. Once listener is running it will listen data source as event source and map incoming data to actual events and then it’ll send it to Herogi Api.
There are two types of listeners available at the moment, file listener and sql listener.
If you have csv or excel files which are generated by other process, you can use it as event source without modifying that file and then you can run a file listener to send that data as events without writing any code.
You can download File Listener from our github repository.
Configuring a file listener is easy, at minimum you need to provide simple event mapping which describes how your data will be represented as events. You can use static configuration like parameter1, parameter2 which sent as it is or you can use index or column name to map a parameter to one of columns 0, 1 or col1, col2.
Listener Configuration
# Name of listener, it will be used in logs
name = "listener"
# Interval to check data directory. Available units seconds/minutes/hours/days
interval = 5 seconds
# Input directory, listener will use this directory as file source
inputPath = /file/path
# Archive directory, once listener process the content of files, it will move file to this path
archivePath = /tmp/
# Offset flust cycle, default is 10
# offsetFlush = 10
Example event mapping configuration:
# Mapping configuration for events. Supported types static/name/index
# Use static for predifened configuration. E.g. registerEvent
# Use index for mapping parameter to specific column in file by index. E.g. 4
# Use name for mapping parameter to specific column in file by header. E.g. username
event-mapper {
# Session id for event
sessionId {
type = name
value = "username"
}
# Scenarios which will get the event
scenarios {
type = static
values = ["scenario1", "scenario2"]
}
# Event name
name {
type = static
value = "registerEvent"
}
# Event parameters as key -> val, keys refer to actual parameter, values refer to equivalent in the file
params {
type = "name"
values = {
p1 = "fullname"
p2 = "age"
}
}
}
To understand better, lets have an example csv file which has content like this;
username,name,age
funkyeng,Chris,29
td,Tyler Durden,34
This will be converted to event after mapping as;
Event 1:
eventName -> registerEvent
scenarios -> scenario1, scenario2
sessionId -> funkeng
parameters -> p1 : Chris, p2: 29
Event 2:
eventName -> registerEvent
scenarios -> scenario1, scenario2
sessionId -> td
parameters -> p1 : Tyler Durden, p2: 34
If you have sql tables used/populated by other processes, you can use it as event source without writing any code. You just need to run a sql listener to send that data as events.
You can download Sql Listener from our github repository.
Configuring a sql listener is easy, you need to provide a sql query to run with an interval. Once you provide a query, sql listeners will run that query on the database which you provided as connection string. Sql listener supports dynamic querying by setting a reference column (datetime or int based). If you provide a condition on these kind of columns, you’ll end up by consuming only new records on each iteration.
For example;
SELECT * FROM users WHERE creation_date > {creation_date} ORDER BY creation_date
This query will give you, only new registered user in that table on each iteration.
Rest of the configuration is similar to file listener, event mapping which describes how your data will be represented as events. You can use static configuration like parameter1, parameter2 which sent as it is or you can use index or column name to map a parameter to one of columns 0, 1 or col1, col2.
Listener Configuration
# Name of listener, it will be used in logs
name = "listener"
# Interval to run query. Available units seconds/minutes/hours/days
interval = 5 seconds
# Sql connection settings
connection {
driver = com.mysql.jdbc.Driver
url = "jdbc:mysql://localhost/yourdb"
user = "username"
password = "12345678"
}
# Query to run, specify sliding conditions by using curly brace, e.g. {creation_date}
query = "SELECT * FROM yourtable"
# Type of sliding query, several modes are available dateColumn | numericColumn | lastSchedule
sliderType = "dateColumn"
# Column to use in sliding queries as reference value
sliderColumn = "creation_date"
# Initial start value for sliding based queries, default is current timestamp
#start = "2009-12-12T00:00:00"
Event mapping configuration;
# Mapping configuration for events. Supported types static/name/index
# Use static for predifened configuration. E.g. registerEvent
# Use index for mapping parameter to specific column in table by index. E.g. 4
# Use name for mapping parameter to specific column in file by header. E.g. username
event-mapper {
# Session id for event
sessionId {
type = name
value = "username"
}
# Scenarios which will get the event
scenarios {
type = static
values = ["scenario1", "scenario2"]
}
# Event name
name {
type = static
value = "registerEvent"
}
# Event parameters as key -> val, keys refer to actual parameter, values refer to equivalent column in the table
# Don't forget to prefix column name with table prefix. E.g. table1_col1, table1_col2, table2_col1
params {
type = "name"
values = {
p1 = "users_fullname"
p2 = "users_age"
}
}
}
To understand better, lets have an example sql table users which has rows like this;
username,fullname,age
funkyeng,Chris,29
td,Tyler Durden,34
This will be converted to event after mapping as;
Event 1:
eventName -> registerEvent
scenarios -> scenario1, scenario2
sessionId -> funkeng
parameters -> p1 : Chris, p2: 29
Event 2:
eventName -> registerEvent
scenarios -> scenario1, scenario2
sessionId -> td
parameters -> p1 : Tyler Durden, p2: 34
If you want to skip event parameter mappings, you can just provide empty block like this;
event-mapper {
# Empty parameter blocks
params {
type = "name"
values = {
}
}
}
This will be converted to event after mapping as;
Event 1:
eventName -> registerEvent
scenarios -> scenario1, scenario2
sessionId -> funkeng
parameters ->
Event 2:
eventName -> registerEvent
scenarios -> scenario1, scenario2
sessionId -> td
parameters ->
If you want to use all available columns as event parameters, you can just remove whole parameter configuration;
event-mapper {
# Session id for event
sessionId {
type = name
value = "username"
}
# Scenarios which will get the event
scenarios {
type = static
values = ["scenario1", "scenario2"]
}
# Event name
name {
type = static
value = "registerEvent"
}
# No parameter configuration provided
}
This will be converted to event after mapping as;
Event 1:
eventName -> registerEvent
scenarios -> scenario1, scenario2
sessionId -> funkeng
parameters -> users_username : funkyeng, users_fullname : Chris, users_age : 29
Event 2:
eventName -> registerEvent
scenarios -> scenario1, scenario2
sessionId -> td
parameters -> users_username : td, users_fullname : Tyler Durden, users_age : 34
Once your listener configuration is completed you can run it by simply ./start.sh on linux systems. Log file should be available as /var/log/herogi/filelistener/application.log
Not: You need to create log directory and give sufficient permissions to user running listener.
Stopping listener is straight forward, you can use ./stop.sh on linux systems. Keep in mind that it will delay the stop till finish current batch.