Don't Repeat Data: Zero Copy
Most systems work by downloading data. You request info from a server, and the server sends a payload back. You then store that data in your own system.
This creates a problem. You end up with duplicate data. You can solve this using a principle called Zero Copy.
What is Zero Copy?
Zero Copy means you access data without moving it. Think of it like opening a window. You see the data through the window, but the data stays in its original home. You do not need to bring the data to your house to look at it.
How ServiceNow uses this:
Operations Management requires data from many places like SAP, AWS, or Snowflake. Usually, you use ETL pipelines to move this data. This takes time and creates errors.
ServiceNow uses Virtual Data Fabric Tables instead. It queries the data only when you ask for it. The system holds the data in memory temporarily. Once you finish your task, the data vanishes.
The benefits:
• You do not duplicate data. • You do not need to sync data between systems. • You reduce security risks because data does not live in a second location.
The trade-offs:
• Speed: You depend on network speed. If the source is slow, you wait. • Uptime: If the source system goes down, you see nothing. You have no local backup. • Load: Every user request sends a live query to the source system. • Costs: Frequent API calls can become expensive. • Reporting: You cannot index this data, so complex reports are harder to run.
When to use Zero Copy:
Use it when moving data costs more than the benefit of storing it.
- Sensitive Data: Keep PII or HIPAA data in one place to reduce risk.
- Real-Time Needs: Use it for IT troubleshooting where stale data is useless.
- Occasional Lookups: Use it to check a single budget number instead of syncing a whole database.
- Massive Datasets: Use it to view summaries of huge logs instead of moving petabytes of data.
Source: https://dev.to/syncrofosatron/dont-repeat-data-zero-copy-1ff0
Optional learning community: https://t.me/GyaanSetuAi