Skip to content

Conversation

dubyte
Copy link
Contributor

@dubyte dubyte commented Aug 8, 2021

The pull request is to try to decode sparkplug in case json fails to parse. it works in the raw format and diff viewers.

@marktoddgea
Copy link

This is awesome! I hope this gets approved ASAP! Much needed.

Copy link
Owner

@thomasnordquist thomasnordquist left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution @dubyte.
Really nice and clean work.

Two things:

  • Do you know where I can get example sparkplugb messages? (I want to write a short test and check that the definition is narrow enough that other valid strings are not interpreted as sparkplug message)
  • ProtobufJS should be able to read and compile .proto files, is there an official source for this proto definition? We could then get rid of the sparkplug.ts file, or generate it during build.
    I just found https://github.com/Cirrus-Link/Sparkplug/blob/master/sparkplug_b/sparkplug_b.proto

Personally I prefer not having generated code in a project if it can be avoided.

Co-authored-by: Thomas Nordquist <thomasnordquist@users.noreply.github.com>
@thomasnordquist
Copy link
Owner

Could be as easy as using the webpack file-loader for .proto files.
I am not entirely sure about the inline-loader syntax.

Adapted from https://github.com/protobufjs/protobuf.js#examples

// Using webpacks file-loader to emit an url to the proto file
import sparkplugBProto from 'file-loader!../../path/to/file/sparkplug-b.proto';

protobuf.load(sparkplugBProto, function(err, root) {
    // Code from the example
    if (err)
        throw err;

    // Obtain a message type
    var AwesomeMessage = root.lookupType("awesomepackage.AwesomeMessage");

    // Exemplary payload
    var payload = { awesomeField: "AwesomeString" };

    // Verify the payload if necessary (i.e. when possibly incomplete or invalid)
    var errMsg = AwesomeMessage.verify(payload);
    if (errMsg)
        throw Error(errMsg);

    // Create a new message
    var message = AwesomeMessage.create(payload); // or use .fromObject if conversion is necessary

    // Encode a message to an Uint8Array (browser) or Buffer (node)
    var buffer = AwesomeMessage.encode(message).finish();
    // ... do something with buffer

    // Decode an Uint8Array (browser) or Buffer (node) to a message
    var message = AwesomeMessage.decode(buffer);
    // ... do something with message

    // If the application uses length-delimited buffers, there is also encodeDelimited and decodeDelimited.

    // Maybe convert the message back to a plain object
    var object = AwesomeMessage.toObject(message, {
        longs: String,
        enums: String,
        bytes: String,
        // see ConversionOptions
    });
});

Webpack

https://webpack.js.org/concepts/loaders/#inline

@dubyte
Copy link
Contributor Author

dubyte commented Aug 11, 2021

That is cool, I will try to do that.

@dubyte
Copy link
Contributor Author

dubyte commented Aug 12, 2021

This is mock so you can test locally: https://github.com/dubyte/sparkplugbmock/releases/tag/v0.1.0

The payload of sparkplugb is binary so it is hard to see: but this is an example:

This one is taking hex string and parsing it to protojson , you can get the decoder from https://github.com/dubyte/sparkplugclidecoder/releases/tag/v0.1.1
echo 08E5ACBDBDB32F12120A056576656E7418E5ACBDBDB32F200B70001800 | sparkplugclidecoder

After xxd that is how it looks in a terminal
echo 08E5ACBDBDB32F12120A056576656E7418E5ACBDBDB32F200B70001800 | xxd -r -p

@dubyte
Copy link
Contributor Author

dubyte commented Aug 12, 2021

Could be as easy as using the webpack file-loader for .proto files.
I am not entirely sure about the inline-loader syntax.

Adapted from https://github.com/protobufjs/protobuf.js#examples

// Using webpacks file-loader to emit an url to the proto file
import sparkplugBProto from 'file-loader!../../path/to/file/sparkplug-b.proto';

protobuf.load(sparkplugBProto, function(err, root) {
    // Code from the example
    if (err)
        throw err;

    // Obtain a message type
    var AwesomeMessage = root.lookupType("awesomepackage.AwesomeMessage");

    // Exemplary payload
    var payload = { awesomeField: "AwesomeString" };

    // Verify the payload if necessary (i.e. when possibly incomplete or invalid)
    var errMsg = AwesomeMessage.verify(payload);
    if (errMsg)
        throw Error(errMsg);

    // Create a new message
    var message = AwesomeMessage.create(payload); // or use .fromObject if conversion is necessary

    // Encode a message to an Uint8Array (browser) or Buffer (node)
    var buffer = AwesomeMessage.encode(message).finish();
    // ... do something with buffer

    // Decode an Uint8Array (browser) or Buffer (node) to a message
    var message = AwesomeMessage.decode(buffer);
    // ... do something with message

    // If the application uses length-delimited buffers, there is also encodeDelimited and decodeDelimited.

    // Maybe convert the message back to a plain object
    var object = AwesomeMessage.toObject(message, {
        longs: String,
        enums: String,
        bytes: String,
        // see ConversionOptions
    });
});

Webpack

https://webpack.js.org/concepts/loaders/#inline

In this example I don't get it, how I can expose Payload if it is created inside a callback?

Sorry I am not used to this syntax

What I want is be able to load the proto during the start of the app, then with an import be able to call decode and fromJson.

I can pass from:

`
// ./backend/src/Model/SparkplugB.ts
const protobuf = require('protobufjs')

// Using webpacks file-loader to emit an url to the proto file, changed webpack config to support proto
import sparkplugBProto from '../../../res/sparkplug_b.proto'

protobuf.load(sparkplugBProto).then(function (root: any) {
const Payload = root.lookupType('com.cirruslink.sparkplug.protobuf.Payload')
})
`

@dubyte
Copy link
Contributor Author

dubyte commented Aug 12, 2021

Yesterday I had been trying to load the proto, but for some reason I can find the type.

// Using webpacks file-loader to emit an url to the proto file
import sparkplugBProto from 'file-loader!./../../../../res/sparkplug_b.proto';   // I put this file in /res 

protobuf.load(sparkplugBProto, function(err, root) {
    // Code from the example
    if (err)
        throw err;

    // Obtain a message type
    var Payload = root.lookupType("com.cirruslink.sparkplug.protobuf.Payload"); // <- the code fails to find out the type :(

@dubyte
Copy link
Contributor Author

dubyte commented Aug 15, 2021

I make it work by loading the proto instead of use a generated code.
I am not sure Payload = undefined is a best practice could you give me a recomendation about that?

Thanks

@dubyte
Copy link
Contributor Author

dubyte commented Aug 23, 2021

@thomasnordquist could you give me a review on this, I did the changes you suggested.
Thanks

@martinscheffler
Copy link

Just wanted to say it is great that you guys are workin on this, this will be very useful indeed!

@marktoddgea
Copy link

Just wanted to say it is great that you guys are workin on this, this will be very useful indeed!

I agree, I have been using MQTT FX because it does decode sparkplugb, but it isn't supported anymore and has some very serious bugs. Sparkplugb is really a great light-weight messaging system and I hope more people start using it. MQTT Explorer is such an outstanding tool. Thanks for all the hard work!

@vdwpsmt
Copy link

vdwpsmt commented Oct 22, 2021

Wow, why didn't I find this mqtt client earlier? awesome this topic tree explorer auto-updating etc...
Any idea when this sparkplug decoding feature could be released?

@dubyte
Copy link
Contributor Author

dubyte commented Jan 5, 2022

Any update about the PR? could you review? Thanks

@danwale
Copy link

danwale commented Jan 24, 2022

I'm also very keen on the Sparkplug B decoding functionality being added, I was previously using MQTT FX but considering it uses a vulnerable version of log4j and isn't getting any updates to the free version anymore I want to move to using this tool instead to do the Sparkplug B decoding easily.

@hobbes1069
Copy link

Adding my +1 to this PR.

@aott33
Copy link

aott33 commented Feb 12, 2022

Adding my +1 as well

@thomasnordquist thomasnordquist merged commit 72400af into thomasnordquist:master Feb 27, 2022
@thomasnordquist
Copy link
Owner

Trying to do a release the next few days.

@hobbes1069
Copy link

Just checking in, if I understand correctly it looks like the CI tests are failing?

@AaronAutomated
Copy link

Bumping this. Would love to see Sparkplug decoding added as a feature as this is by far the best MQTT client for testing and development.

@Ekion-1
Copy link

Ekion-1 commented Jul 11, 2022

Agreed. This would be a game changer.

@tord-v
Copy link

tord-v commented Sep 23, 2022

Bumping. All I want for Christmas is this.

@iansebryk
Copy link

shameless bump. this would alter the landscape forever. please keep going on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.