{"id":325,"date":"2022-04-22T00:00:00","date_gmt":"2022-04-22T00:00:00","guid":{"rendered":"https:\/\/tac.debuzzify.com\/?p=325"},"modified":"2023-06-27T06:29:57","modified_gmt":"2023-06-27T06:29:57","slug":"data-team-structure-centralized-embedded-hybrid","status":"publish","type":"post","link":"https:\/\/www.the-analytics.club\/data-team-structure-centralized-embedded-hybrid\/","title":{"rendered":"Data Teams Are Becoming Less Centralized, and That&#8217;s Wonderful"},"content":{"rendered":"\n<p>I firmly believe there won\u2019t be any data teams in the future.<\/p>\n\n\n\n<p>We all have witnessed the birth and growth of data teams in the past decades. And I belong in one of them.<\/p>\n\n\n\n<p>But we will have to see the disappearance of it too!<\/p>\n\n\n\n<p>Yet that\u2019s something the data science community should celebrate.<\/p>\n\n\n\n<p>Traditionally, every organization has dedicated units for every aspect of its business\u200a\u2014\u200afinance, planning, etc. That\u2019s how we thought data science, too, deserves a dedicated team.<\/p>\n\n\n\n<p>Fair enough, it works well so far.<\/p>\n\n\n\n<p>Yet, we also see most people with no STEM background getting into AI\/ML. Not to mention, they do well in their jobs, just like anyone else.<\/p>\n\n\n\n<p>As we advanced in uncovering new ways to use our data, we\u2019ve also built great tools that enable anyone to become a data scientist.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">The evolution of modern data&nbsp;science<\/h3>\n\n\n\n<p>When I started in data science, the most significant achievement at the time was <b>low-code<\/b> libraries.<\/p>\n\n\n\n<p>Take <a href=\"https:\/\/scikit-learn.org\/stable\/\" target=\"_blank\" rel=\"noopener\">scikit-learn<\/a>, for example. It masks the implementation of how an algorithm is implemented. All the data scientists will have to worry about is its application.<\/p>\n\n\n\n<p>For instance, to build a linear regression model for your data set, you import the LinearRegression function and call it with our dependent and independent variables, like in the example below.<\/p>\n\n\n\n<p><script><br \/>\n            \/* This is ugly, but to ensure we only create this function once, and only call<br \/>\n            each JS library once, we need to check here in case there are multiple Wagtail<br \/>\n            blocks on this page. This will ensure we only load the minimum payload. *\/<br \/>\n            if(typeof loadPrismLanguage != 'function') {<br \/>\n                window.loadPrismLanguage = function(language) {<br \/>\n                    var libraries = [<br \/>\n                        {<br \/>\n                            \"id\": \"code-block-prismjs\",<br \/>\n                            \"url\": \"\/\/cdnjs.cloudflare.com\/ajax\/libs\/prism\/1.25.0\/prism.min.js\"<br \/>\n                        },<br \/>\n                        {<br \/>\n                            \"id\": \"code-block-prismjs-\" + language,<br \/>\n                            \"url\": \"\/\/cdnjs.cloudflare.com\/ajax\/libs\/prism\/1.25.0\/components\/prism-\" + language + \".min.js\"<br \/>\n                        },<br \/>\n        {<br \/>\n            \"id\": \"code-block-line-numbers\",<br \/>\n            \"url\": \"\/\/cdnjs.cloudflare.com\/ajax\/libs\/prism\/1.25.0\/plugins\/line-numbers\/prism-line-numbers.min.js\"<br \/>\n        }<\/p>\n<p>                    ];<\/p>\n<p>                    for(const library of libraries) {<br \/>\n                        if(document.getElementById(library[\"id\"]) == null) {<br \/>\n                            var s = document.createElement(\"script\");<br \/>\n                            s.id = library[\"id\"];<br \/>\n                            s.type = \"text\/javascript\";<br \/>\n                            s.src = library[\"url\"];<br \/>\n                            s.async = false;<br \/>\n                            document.body.appendChild(s);<br \/>\n                        }<br \/>\n                    }<br \/>\n                };<br \/>\n            }<\/p>\n<p>            loadPrismLanguage('python');<\/p>\n<p>            language_class_name = 'language-python';<br \/>\n            <\/script><\/p>\n\n\n\n<div class=\"wp-block-kevinbatdorf-code-block-pro padding-bottom-disabled\" data-code-block-pro-font-family=\"Code-Pro-JetBrains-Mono\" style=\"font-size:16px;font-family:Code-Pro-JetBrains-Mono,ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace;line-height:20px\"><span style=\"display:block;padding:16px 0 0 16px;margin-bottom:-1px;width:100%;text-align:left;background-color:#2e3440ff\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"54\" height=\"14\" viewBox=\"0 0 54 14\"><g fill=\"none\" fill-rule=\"evenodd\" transform=\"translate(1 1)\"><circle cx=\"6\" cy=\"6\" r=\"6\" fill=\"#FF5F56\" stroke=\"#E0443E\" stroke-width=\".5\"><\/circle><circle cx=\"26\" cy=\"6\" r=\"6\" fill=\"#FFBD2E\" stroke=\"#DEA123\" stroke-width=\".5\"><\/circle><circle cx=\"46\" cy=\"6\" r=\"6\" fill=\"#27C93F\" stroke=\"#1AAB29\" stroke-width=\".5\"><\/circle><\/g><\/svg><\/span><span role=\"button\" tabindex=\"0\" data-code=\"from sklearn.linear_model import LinearRegression\nreg = LinearRegression().fit(X, y)\" style=\"color:#d8dee9ff;display:none\" aria-label=\"Copy\" class=\"code-block-pro-copy-button\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"width:24px;height:24px\" fill=\"none\" viewBox=\"0 0 24 24\" stroke=\"currentColor\" stroke-width=\"2\"><path class=\"with-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2m-6 9l2 2 4-4\"><\/path><path class=\"without-check\" stroke-linecap=\"round\" stroke-linejoin=\"round\" d=\"M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2\"><\/path><\/svg><\/span><pre class=\"shiki nord\" style=\"background-color: #2e3440ff\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color: #81A1C1\">from<\/span><span style=\"color: #D8DEE9FF\"> sklearn<\/span><span style=\"color: #ECEFF4\">.<\/span><span style=\"color: #D8DEE9FF\">linear_model <\/span><span style=\"color: #81A1C1\">import<\/span><span style=\"color: #D8DEE9FF\"> LinearRegression<\/span><\/span>\n<span class=\"line\"><span style=\"color: #D8DEE9FF\">reg <\/span><span style=\"color: #81A1C1\">=<\/span><span style=\"color: #D8DEE9FF\"> <\/span><span style=\"color: #88C0D0\">LinearRegression<\/span><span style=\"color: #ECEFF4\">().<\/span><span style=\"color: #88C0D0\">fit<\/span><span style=\"color: #ECEFF4\">(<\/span><span style=\"color: #D8DEE9FF\">X<\/span><span style=\"color: #ECEFF4\">,<\/span><span style=\"color: #D8DEE9FF\"> y<\/span><span style=\"color: #ECEFF4\">)<\/span><\/span><\/code><\/pre><span style=\"display:flex;align-items:flex-end;padding:10px;width:100%;justify-content:flex-end;background-color:#2e3440ff;color:#c8d0e0;font-size:12px;line-height:1;position:relative\">Python<\/span><\/div>\n\n\n\n<p><script><br \/>\n                var block_num = (typeof block_num === 'undefined') ? 0 : block_num;<br \/>\n                block_num++;<br \/>\n                document.getElementById('target-element-current').className = language_class_name;<br \/>\n                document.getElementById('target-element-current').id = 'target-element-' + block_num;<br \/>\n            <\/script><\/p>\n\n\n\n<p>More libraries <a href=\"https:\/\/keras.io\/\" target=\"_blank\" rel=\"noopener\">Keras<\/a> and <a href=\"https:\/\/pytorch.org\/\" target=\"_blank\" rel=\"noopener\">PyTorch<\/a>, are extended the same philosophy to neural networks and deep learning too.<\/p>\n\n\n\n<p>We still had to have programming skills to work with low-code tools. Later came the <b>visual analytics<\/b> platforms such as <a href=\"https:\/\/www.knime.com\/\" target=\"_blank\" rel=\"noopener\">KNIME<\/a>. They allowed us to use sophisticated data science techniques without writing a single line of code.<\/p>\n\n\n\n<p><a href=\"\/become-a-data-scientist-or-data-engineer-without-coding-skills\"><i>Making sense from data is common sense<\/i><\/a><i>. Coding skills aren\u2019t the superpower of a data scientist or data engineer.<\/i><\/p>\n\n\n\n<p>That was amazing.<\/p>\n\n\n\n<p>Visual analytics has several benefits compared to low-code libraries.<\/p>\n\n\n\n<p>Especially those who were starting in data science find it helpful. The drag-and-drop implementation of the data science pipeline helped them grasp the core concepts faster.<\/p>\n\n\n\n<p>Even experienced data scientists find it useful for rapid prototyping and a shorter time to go live.&nbsp;<\/p>\n\n\n\n<p>It\u2019s not the end; we\u2019ve got <b>AutoML<\/b>!<\/p>\n\n\n\n<p>I wouldn\u2019t call it a drawback, but we still have to know about different algorithms and their internals to use visual analytics.<\/p>\n\n\n\n<p>It leads us to AutoML\u200a\u2014\u200aAI figures out which AI to work with your data.<\/p>\n\n\n\n<p>Auto ML platforms help everyone to solve business problems with AI. It solves some pre-defined problems with arbitrary data from the user.<\/p>\n\n\n\n<p><b>Related:<\/b> <a href=\"\/data-science-will-be-democratized\"><b><i>Data Science Will Be Democratized (In Less Than 10 Years.)<\/i><\/b><\/a><\/p>\n\n\n\n<p>For instance, you can perform a churn prediction by uploading a dataset to <a href=\"https:\/\/aws.amazon.com\/sagemaker\/canvas\/\" target=\"_blank\" rel=\"noopener\">SageMaker Canvas<\/a> and running a regression model over it. You don\u2019t have to do feature selection, hyperparameter tuning, and other time-consuming technical work.<\/p>\n\n\n\n<p>In minutes you get your model trained and ready for prediction. You can even <a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/deploy-model.html\" target=\"_blank\" rel=\"noopener\">deploy it as a prediction service<\/a> (REST API.)<\/p>\n\n\n\n<p><b>Related:<\/b> <a href=\"https:\/\/towardsdatascience.com\/3-ways-to-deploy-machine-learning-models-in-production-cdba15b00e\" target=\"_blank\" rel=\"noopener\"><b><i>3 Ways to Deploy Machine Learning Models in Production<\/i><\/b><\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How is the role of data scientists becoming more embedded?<\/h3>\n\n\n\n<p>Centralized data science teams are very productive. Such units offer incredible opportunities for their members to learn new skills quickly.<\/p>\n\n\n\n<p>\u00a0<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"631\" height=\"354\" src=\"https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/image-18.png\" alt=\"Traditional centralized data team\" class=\"wp-image-717\" title=\"\" srcset=\"https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/image-18.png 631w, https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/image-18-300x168.png 300w\" sizes=\"(max-width: 631px) 100vw, 631px\" \/><\/figure><\/div>\n\n\n<p>&nbsp;<\/p>\n\n\n\n<p>Also, the management can prioritize critical resources at the organizational level. It makes sure (at least from the management\u2019s perspective) that the ROI is at its greatest.<\/p>\n\n\n\n<p>A drawback of centralized data teams is that they are often disconnected from the actual business problem. For instance, the prediction accuracy of drug efficacies has to be higher than the market demand prediction accuracy.<\/p>\n\n\n\n<p><b>Related:<\/b> <a href=\"https:\/\/towardsdatascience.com\/the-1-mistake-companies-make-when-creating-their-data-science-foundation-ff775d9e20b1\" target=\"_blank\" rel=\"noopener\"><b><i>The #1 Mistake Companies Make When Creating Their Data Science Foundation<\/i><\/b><\/a><\/p>\n\n\n\n<p><b>Related:<\/b> <a href=\"https:\/\/towardsdatascience.com\/stop-firefighting-data-quality-issues-4c064bcbbb53\" target=\"_blank\" rel=\"noopener\"><b><i>Stop Firefighting Data Quality Issues.<\/i><\/b><\/a><\/p>\n\n\n\n<p>In this example, it may be the person\u2019s common sense to choose a higher threshold for drug efficacy. But real-life data science problems require tons of such domain expertise for a model to be useful.<\/p>\n\n\n\n<p>In centralized teams, we always have a risk of business disconnection. There is no better solution than having a dedicated data scientist(s). In other words, make the data team decentralized and embedded.<\/p>\n\n\n\n<p>\u00a0<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"631\" height=\"354\" src=\"https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/image-19.png\" alt=\"Embedded data teams\" class=\"wp-image-718\" title=\"\" srcset=\"https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/image-19.png 631w, https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/image-19-300x168.png 300w\" sizes=\"(max-width: 631px) 100vw, 631px\" \/><\/figure><\/div>\n\n\n<p>The challenge in decentralizing data scientists is limited communication with other data scientists. It reduces opportunities for collaborative learning. The results are poor data standards across the organization and slow problem-solving.<\/p>\n\n\n\n<p>This is where I see the real benefit of Auto ML and visual analytics.<\/p>\n\n\n\n<p>In other words, it is now possible to extract the application layer and move it closer to <a href=\"https:\/\/www.the-analytics.club\/data-leakage-and-its-business-impacts\/\" title=\"Data Leakage and its Consequences for Businesses\">business units in the entire data science workflow<\/a>.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote\">\n<p>In the data science workflow, it is now possible to extract the application layer and move it closer to business&nbsp;units.<\/p>\n<\/blockquote>\n\n\n\n<p>Each team will have either a <a href=\"https:\/\/www.the-analytics.club\/welcome-to-the-age-of-citizen-data-scientists\/\" title=\"Welcome to the Age of Citizen Data Scientists\">dedicated data scientist<\/a> or someone trained in data science in this new approach. This person could now build ml models and bi-dashboards on top of the organizational <a href=\"https:\/\/www.the-analytics.club\/data-warehouse-data-lake-lakehouse\">data warehouse\/data lake<\/a>.<\/p>\n\n\n\n<p>The benefit here is this person is more domain expertise than the one in the centralized team.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Well, not that decentralized!<\/h3>\n\n\n\n<p>Some tasks in the data science workflow have nothing much to do with the subject matter.<\/p>\n\n\n\n<p>Take the case of a data engineer. A data engineer\u2019s work is to get the right data into the data warehouse. It has very little to do with decisions such as the model to be used, model accuracy, etc.<\/p>\n\n\n\n<p>Embedding such roles to individual business units can do more harm than good. Instead, they could work in a centralized data team.<\/p>\n\n\n\n<p>Thus, what\u2019s more relevant to modern organizations is a <a href=\"https:\/\/mikkeldengsoe.substack.com\/p\/data-team-structure-embedded-or-centralised\" target=\"_blank\" rel=\"noopener\">hybrid approach<\/a>.<\/p>\n\n\n\n<p>\u00a0<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"631\" height=\"354\" src=\"https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/image-20.png\" alt=\"Hybrid data teams\" class=\"wp-image-719\" title=\"\" srcset=\"https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/image-20.png 631w, https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/image-20-300x168.png 300w\" sizes=\"(max-width: 631px) 100vw, 631px\" \/><\/figure><\/div>\n\n\n<p>Roles such as data engineer or anything that has little relevance to the application side of data science could go into a centralized data team. Roles that closely work with business units go well embedded.<\/p>\n\n\n\n<p>Having a centralized team for data engineering also ensures common data standards are used across the organization. This hybrid approach is no more a problem for embedded data teams.&nbsp;<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Final thoughts<\/h3>\n\n\n\n<p>Data science teams have discovered new heights in the past couple of decades. Along with it, we discovered new roles and responsibilities too.&nbsp;<\/p>\n\n\n\n<p>Traditionally, data teams worked in isolation as a centralized entity. It serves other business units in the organization. But the data team itself is driven by its own set of standards and KPIs.&nbsp;<\/p>\n\n\n\n<p>However, recent advancements in data science, such as Auto ML and visual analytics, make data science skills available to everyone. You don\u2019t have formal training to build and deploy machine learning models.&nbsp;<\/p>\n\n\n\n<p>Hence, most business problems can be solved by their respective units themself without needing a data scientist.&nbsp;<\/p>\n\n\n\n<p>Only common tasks such as data engineering remain a central role in the organization.\u00a0<\/p>\n\n\n\n<hr class=\"wp-block-separator has-text-color has-cyan-bluish-gray-color has-alpha-channel-opacity has-cyan-bluish-gray-background-color has-background is-style-dots\"\/>\n\n\n\n<p>Thanks for reading, friend! Say Hi to me on <a href=\"https:\/\/www.linkedin.com\/in\/thuwarakesh\/\" target=\"_blank\" rel=\"nofollow noopener\"><b>LinkedIn<\/b><\/a>, <a href=\"https:\/\/twitter.com\/Thuwarakesh\" target=\"_blank\" rel=\"nofollow noopener\"><b>Twitter<\/b><\/a>, and <a href=\"https:\/\/thuwarakesh.medium.com\/\" target=\"_blank\" rel=\"nofollow noopener\"><b>Medium<\/b><\/a>.<\/p>\n\n\n\n<p>Not a Medium member yet? Please use this link to become a member because, at no extra cost to you, I earn a small commission for referring you.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Businesses should celebrate modern shifts in the AI\/ML technology landscape.<\/p>\n","protected":false},"author":2,"featured_media":186,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_kad_blocks_custom_css":"","_kad_blocks_head_custom_js":"","_kad_blocks_body_custom_js":"","_kad_blocks_footer_custom_js":"","_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"footnotes":""},"categories":[10],"tags":[],"taxonomy_info":{"category":[{"value":10,"label":"Strategy"}]},"featured_image_src_large":["https:\/\/www.the-analytics.club\/wp-content\/uploads\/2023\/06\/pexels-laura-tancredi-7078666-1024x683.webp",1024,683,true],"author_info":{"display_name":"Thuwarakesh","author_link":"https:\/\/www.the-analytics.club\/author\/thuwarakesh\/"},"comment_info":0,"category_info":[{"term_id":10,"name":"Strategy","slug":"opinion-strategy","term_group":0,"term_taxonomy_id":10,"taxonomy":"category","description":"","parent":0,"count":32,"filter":"raw","cat_ID":10,"category_count":32,"category_description":"","cat_name":"Strategy","category_nicename":"opinion-strategy","category_parent":0}],"tag_info":false,"_links":{"self":[{"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/posts\/325"}],"collection":[{"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/comments?post=325"}],"version-history":[{"count":3,"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/posts\/325\/revisions"}],"predecessor-version":[{"id":1297,"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/posts\/325\/revisions\/1297"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/media\/186"}],"wp:attachment":[{"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/media?parent=325"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/categories?post=325"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.the-analytics.club\/wp-json\/wp\/v2\/tags?post=325"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}