Will data engineering be automated in future?

Kaustubh Katdare

Kaustubh Katdare

@thebigk
Updated: Oct 26, 2024
Views: 1.5K

No, data engineering will not be 100% automated in future. However, several aspects of data engineering will be aided by AI; but still require human intervention.

The AI has seen unbelievable advances in recent times and people are worrying if AI robots will replace the data engineering in future. The data engineering roles are specialists roles.

Replaced by robots - a website keeps track of the jobs that are being replaced, and will be replaced in future suggests that the chances of automation in data engineering are very slim: about 2.9%. This clearly indicates that data engineering won't be fully automated at least in the next 10-15 years.

Data engineering involves collection, validation, conditioning and maintenance. Each of these steps requires a specific skill-set that's better applied by the humans. The amount of subjective decision making involved in the process is an indicator that AI won't come after data engineers easily.

Let me know your thoughts. Do you think data engineers need to worry about getting their jobs replaced by AI?

Like 0 Replies 1

Replies

Welcome, guest

Join CrazyEngineers to reply, ask questions, and participate in conversations.

Sign up Log in

CrazyEngineers powered by Jatra Community Platform

  • Kaustubh Katdare

    Kaustubh Katdare

    @thebigk • Feb 15, 2023

    Will AI replace data engineers?

    AI will not fully replace data engineers. As discussed, only certain components of the entire data engineering workflow can be automated. Look at the typical functions of data engineers -

    1. Prepare data, verifying it, cleaning it etc.
    2. Build data pipelines to migrate data from source to destination; which could be a cloud on on-prem server.
    3. Deploying pipelines at client sites and/or on the cloud.

    AI can take care of some parts - like verification, cleaning of data. But for mission critical projects, human intervention will be needed.

    Data engineers will adopt AI to automate certain tasks under their observation. DE will see a good inclusion of AI in several of the workflows; and engineers need to ensure that they are aware of latest trends and tools in this domain.