Working on Instill’s pipeline-backend project was like solving a jigsaw 🧩 puzzle—except some pieces kept changing names! My mission? To create a component that could rename JSON fields without creating conflicts. Join me as I share my journey through learning Go, studying Instill’s well-organized
pipeline-backend manages all pipeline resources within for more
details.
Recipe
A pipeline recipe specifies how components are configured and how they are
interconnected.
Recipes are defined in YAML language:
variable
# pipeline input fields
output:
# pipeline output fields
component:
<component-id>:
type: <component-definition-id>
task: <task-id>
input:
# values for the input fields
condition: <condition> # conditional statement to execute or bypass the…Once I got comfortable, ChunHao, who had crafted a
Step 3️⃣: Building the Component
Armed with coffee☕ and courage💪, I got down to coding. Here’s a sneak peek at the core logic:
Mapping Fields
First, I created a mapping system to track old and new field names. This was key to detecting conflicts.
func mapFields(fields map[string]string) map[string]string {
newFieldMap := make(map[string]string)
for oldName, newName := range fields {
// Check for conflict
if _, exists := newFieldMap[newName]; exists {
newName += "_conflict" // Add suffix for conflicts
}
newFieldMap[oldName] = newName
}
return newFieldMap
}
Any time a conflict was detected, the function added "_conflict" to the new name. It’s a simple trick that ensures our JSON fields stay unique and, most importantly, friendly to each other! ✌️
Renaming Fields
Once the field mappings were in place, the next step was applying them to our JSON data.
func renameFields(data map[string]interface{}, fieldMap map[string]string) map[string]interface{} {
renamedData := make(map[string]interface{})
for key, value := range data {
newKey := fieldMap[key]
renamedData[newKey] = value
}
return renamedData
}
Here’s the logic that applies the mapped names to our JSON data. The result? Our data’s neatly renamed, conflicts resolved, and structure intact. 🔥
After creating the component dropped the draft PR & got a comment:
Testing time! 🧪 I wrote tests covering everything from simple renames to complex edge cases with nested JSON fields. Each round of testing led to further refinements.
{
name: "ok - rename fields with overwrite conflict resolution",
// Test where renaming conflicts are resolved by overwriting the existing field.
// Expected outcome: "newField" holds "value1", overwriting any previous value.
},
{
name: "ok - rename fields with skip conflict resolution",
// Test where conflicts are resolved by skipping the rename if the target field already exists.
// Expected outcome: "newField" remains "value2" without overwrite.
},
{
name: "nok - rename fields with error conflict resolution",
// Test with "error" strategy, which should raise an error on conflict.
// Expected outcome: Error message "Field conflict."
},
// Additional cases for missing required fields and invalid conflict resolution strategy
Here’s where I’d love to share a personal reflection: Testing was the hardest part of this project 😮💨. There were times when I thought, "Is this test even doing what it’s supposed to?"
Just then, I ran into a lint issue—
That’s when I realized I had to run the tests locally before submitting. ChunHao also added:
"Please run and pass it before you request the review. Run
$ go test ./pkg/component/operator/json/v0/...to check it locally."
I quickly ran the tests locally, identified the issues, and fixed them.
A little moment of celebration 🥳
This process made me appreciate the importance of local testing even more, as it ensured everything was solid before submitting for review.
Before merging, ChunHao did a final review, made a few tweaks, QAed Test Recipe and updated the documentation to reflect the new changes. Big thanks to Anni for her ongoing support throughout the process—it made a huge difference. 🙌
👥 Reflection on the Collaborative Process 🫱🏼🫲🏼
One of the biggest lessons I learned was how collaboration and mentorship can make or break a project. Instill's moderators, Anni and ChunHao, provided me with the guidance I needed when I was lost in Go syntax or struggling with the right approach. Working together, we turned a complex problem into a clean, functional solution.
I’ll be honest, there were moments I felt like I had bitten off more than I could chew. But the constant encouragement from Anni, combined with the clear direction from ChunHao, kept me on track.
⏭️ Next Steps and Future Improvements
Another step could be expanding this approach to other parts of the pipeline that require dynamic field name handling—because who doesn’t love a little bit of automation⚙️?
🛠️ Tools & Resources 🪛
: A goldmine of well-organized resources to understand the Instill pipeline.
: A Go linters aggregator to identify issues and enforce code quality during development and CI checks.
🧑🏻💻🗒️ My Learning
With Instill’s rock-solid documentation, guidance from ChunHao, and Anni's moral support, this project became a fantastic learning experience. I went from knowing nothing about Go to implementing a fully functional feature ready for production (and I have the merged PR to prove it 😉).
Proof:
Because
- we want to manipulate JSON data
This commit
- provide the function to rename the json key with different strategies.
Community-Analysen & Experten-Meinungen 0
Verwandte Story-Cluster & Quellen (Vektor-KI)
Ähnliche Beiträge
Auch interessante Nachrichten From Zero to Merge: Building a JSON Renaming Field Component in Go
Thematisch verwandte Begriffe: From, Zero, Merge, Building · 6 Treffer
Apple accuses OpenAI of destroying evidence as trade-secrets fight intensifies
GPT-6 Astra Release Today? OpenAI’s Next Major AI Model Is Almost Here
Videos werden geladen ...
Beiträge werden geladen ...
Videos werden geladen ...
Beiträge werden geladen ...
Videos werden geladen ...
Beiträge werden geladen ...
Videos werden geladen ...
Beiträge werden geladen ...
Videos werden geladen ...
posted on to implement the functions.
JSON schema pseudo code
Key Features:
conflictResolution: Handling conflicts when renaming fields in JSON, especially when working with nested objects and dot notation, is critical to avoid data loss or unexpected behavior. Allow users to specify how they want conflicts to be resolved (e.g., via a parameter such asconflictResolution: 'overwrite'|'skip'|'error'),Here are different strategies to manage conflicts and some considerations for each.
1. Overwrite the Existing Field (Default Behavior)
Description: If the
newFieldalready exists in the object, overwrite its value with the value fromcurrentField.Pros:
Cons:
Implementation:
2. Skip the Renaming Operation
Description: If the
newFieldalready exists, skip the renaming operation for that particular field.Pros:
Cons:
Implementation:
3. Merge Values
Description: If both
currentFieldandnewFieldexist and contain objects or arrays, merge the two values. This approach is more complex but can be very powerful.Pros:
Cons:
currentFieldandnewFielddiffer.Implementation:
4. Rename with a Suffix or Prefix
Description: If the
newFieldalready exists, rename the new field by appending a suffix or prefix (e.g.,_1,_conflict) to avoid conflicts.Pros:
Cons:
Implementation:
5. Return an Error or Warning
Description: If a conflict is detected, stop the operation and return an error or warning to the user. This forces the user to address the conflict before proceeding.
Pros:
Cons:
Implementation:
Summary:
Overwrite: Simple and effective, but can lead to data loss.
Skip: Safe but may leave data unchanged.
Error/Warning: Forces user intervention; best for critical operations.
Choose the strategy that best aligns with your application's needs and the user's expectations. Implementing a combination of these strategies, such as providing a default behavior with options for customization, can offer the best balance between usability and robustness.
Example Usage:
Scenario: Input data as JSON object
// input
{
"data": {
"name": "John Doe",
"age": 30,
"address": {
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
},
"state": "conflict"
},
"fields": [
{"currentField": "address.street", "newField": "address.road"},
{"currentField": "state", "newField": "address.state"}
],
// "supportDotNotation": true,
"conflictResolution": "overwrite"
}
Conflict Resolution Scenarios:
1. Overwrite (Default):
2. Skip:
3. Error:
address.statealready exists.ValueError: Conflict detected: 'address.state' already exists.Scenario: Input Data as an Array of Objects
If the input data is an array of objects, the logic needs to be adapted to handle each object in the array individually. The schema and the function would process each object within the array according to the specified fields and
conflictResolutionrules.Below is an example demonstrating how the "Rename Fields" operation would work with input data that is an array of objects.
Input
Explanation:
Output:
Rules for the Component Hackathon