Sources

What are Sources and how to use them?

Sources provide connectivity to the systems that Drasi can observe as sources of change. Sources perform three important functions within Drasi:

  • Process the change log/feed generated by the source system and push those changes to each Continuous Query that uses that Source as input.
  • Translate source change data into a consistent property graph data model so that subscribed Continuous Queries can use that data as if it where a graph of Nodes and Relations. For graph sources, such as Gremlin, no translation is necessary. But for non-graph sources, such as PostgreSQL and Kubernetes, the Source transforms the data (more detail is provided in the individual Sources sections below).
  • Provide a way for Continuous Queries to query the source system at startup to initialize the state of the Continuous Query result.
End to End

Creation

Sources can be created and managed using the Drasi CLI.

The easiest way to create a Source, and the way you will often create one as part of a broader software solution, is to:

  1. Collect endpoint addresses and credentials that provide access to the change log and query API of the source system you want to connect to.
  2. Create Kubernetes secrets containing the credentials the Source will use to connect to the external data source.
  3. Create a YAML file containing the Source resource definition. This will include the configuration settings that enable the Source to connect to the source system and refer to to the Kubernetes secrets that contain the credentials it should use. This file can be stored in your solution repo and versioned along with all the other solution code / resources.
  4. Run drasi apply to apply the Source resource definition to your Drasi environment.

As soon as the Source is created it will start running, monitoring its source system for changes and pushing them to subscribed Continuous Queries.

The definition for a Source has the following basic structure:

apiVersion: v1
kind: Source
name: <id>
spec:
  kind: <source-type>
  properties:
    (source kind specific fields)...

The following table describes these configuration settings:

Name Description
apiVersion Must have the value v1
kind Must have the value Source
name Provides the unique ID of the Source. This is used to manage the Source and in Continuous Query definitions to configure which Sources the Continuous Query uses as input.
spec.kind The type of Source to create, which must be one of the available Source Providers registered in the Drasi environment.
spec.properties Everything in the spec.properties section is unique to the type of Source you are creating. See the individual page for the Source you are creating for details.

Note that any of the properties in the spec.properties section can either be specified inline or reference a Kubernetes secret. eg.

apiVersion: v1
kind: Source
name: my-db
spec:
  kind: PostgreSQL
  properties:
    user: my-user
    password:
      kind: Secret
      name: pg-creds
      key: password

Once configured, to create a Source defined in a file called source.yaml, you would run the command:

drasi apply -f source.yaml

You can then use the drasi list commands to query the existence and status of the Source. For example, to see a list of the active Sources, run the following command:

drasi list source

Deletion

To delete an active Source, run the following drasi delete command:

drasi delete source <id>

For example, if the Source ID is human-resources, you would run,

drasi delete source human-resources

Note: Drasi does not currently enforce dependency integrity between Sources and Continuous Queries. If you delete a Source that is used by one or more Continuous Queries, they will stop getting change events and stop producing results.

Available Sources

Drasi currently provides Sources for the following source systems: