Scaling data operations isn’t easy, but most obstacles are predictable and avoidable. After speaking with current or ex-Head of Data/CTO/CEO or CPO from companies like La Belle Vie, BSport, Qare, Sunday or LeBonCoin, one thing is clear: the biggest challenges are often self-inflicted. From messy governance to tech stacks that don’t scale, let’s skip the fluff and dive straight into what works (and what doesn’t).

1. Centralize now, specialize later

Don’t overthink it. Early on, you need a small, centralized team that can enforce governance and standardize processes. Skip the urge to hire specialists. Get the basics right first — clean pipelines, solid reporting and clear ownership. When the time comes to expand, specialize your team around key business needs.

⚠️ Trap to avoid: Creating silos too early by scattering specialists across teams

💡 Actionable tip: Start with generalists using flexible tools like dbt or Looker. Specialize only when your operational load demands it

2. Pick a tech stack that scales

Your tools should never hold you back. If you’re still using outdated solutions, you’re already behind. The modern data stack is cloud-based, modular, and designed for growth. Think Snowflake, BigQuery, or Databricks. Stop patching systems together and invest in tech that’ll save you headaches down the road.

⚠️Trap to avoid: Locking yourself into rigid, legacy systems

💡Actionable tip: Map out where your business will be in three years. Choose tools that can handle 10x the data volume you’re working with today

3. Governance is non-negotiable

It’s tempting to leave governance for “later.” Don’t, please. By the time you realize you need it, it’ll be too late. Strong governance ensures trust, security, and compliance. Start small: standardized data definitions, access controls, and audit trails. Build from there.

⚠️Trap to avoid: Scrambling to retrofit governance when regulations hit

💡Actionable tip: Implement automated tools like Monte Carlo or CastorDoc to monitor data quality and compliance from day one

4. Data must serve business goals

If your data isn’t driving decisions, what’s the point? Forget dashboards no one reads or non-actionable so-called KPIs with no impact. Your data should be laser-focused on solving business-critical problems. Tie every data initiative to clear, measurable outcomes.

⚠️Trap to avoid: Becoming a request factory for ad-hoc requests

💡Actionable tip: Say no to low-value. Instead, build scalable solutions that answer recurring questions

5. Balance your team

Too many juniors, and progress slows to a crawl. Too many seniors, and your budget implodes. The key is balance. Juniors need mentorship; seniors need to stay focused on high-impact tasks. Keep your team lean but effective.

Trap to avoid: Thinking more headcount means more output

💡Actionable tip: Pair every junior hire with a senior mentor to fast-track learning and autonomy

No excuses. Scale smart

Scaling isn’t about luck; it’s about preparation. The biggest mistakes happen when teams try to rush without a solid foundation. Centralize first, invest in scalable tech, enforce governance, and focus on business impact. If you’re not doing this, your data operations aren’t ready to scale. 

In today's data-driven world, accurately categorizing information has become crucial. The task of multi-label classification, where data can belong to multiple categories simultaneously, addresses this need effectively. Unlike traditional single-label classification, multi-label classification mirrors the complexity of real-world data and often comes to the forefront in wider data analysis efforts. This problem is particularly critical in areas such as social media tagging, news organization, disease diagnosis, and product recommendation. It becomes especially challenging in text analysis, where a single span of text can reference multiple notions in nuanced ways.

While traditional Natural Language Processing (NLP) techniques have been highly effective for many tasks, they often require substantial domain knowledge and manual feature engineering. Advances in deep learning and the emergence of Large Language Models (LLMs) have shifted the landscape, offering more powerful and automated solutions for complex NLP problems, as well as a more precise and insightful approach to text analysis.

TL;DR

Multi-label classification, crucial for tasks like social media tagging, is greatly enhanced by LLMs. These models automate the handling of complex, real-world data, making them ideal when traditional model training is too complex. With user-friendly APIs, LLMs offer cost-effective, high-performance solutions for text analysis. Techniques like few-shot prompting and chain of thought further improve their accuracy. Regular monitoring and evaluation ensure these models stay effective in production.

A quick note on Multi Label Classification

In multi-label classification, unlike multi-class classification where each instance is assigned to exactly one out of many classes, each instance can be assigned to zero, one, or more classes. This means that each class label is not mutually exclusive, and the number of labels per instance isn't fixed. Indeed, each instance in the dataset can have one or more labels assigned to it, or potentially none at all. As such, there can be relationships or dependencies among the labels, meaning the presence of one label might influence the presence of another.

These labels can be represented in various ways:

• One column per category containing binary indicators (0 or 1) showing whether the label is applicable to the document ;
• A list of labels associated with the text content, stored in a single column.

This type of problem is particularly challenging because as the number of possible labels increases, the label matrix becomes more sparse. This means there may be very few examples for many label combinations, making it difficult for models to learn effectively. Typically, a large amount of data is required for model training in such cases.

But what should you do when you don’t have the data to tackle multi labels classification problems in a text setting or you don't have the resources to build it ?

Why Should You Choose LLMs Over Traditional NLP Techniques for Text Analysis?

Traditionally, NLP project involve multiple phases, such as:

• Preprocessing: converting all text to lowercase, removing punctuation, special characters and numbers, reducing words to their base form through stemming or lemmatization
• Feature Extraction: using Bag of Words or N-grams techniques
• Model Selection and Hyper Parameter tuning

The specific case of multi label classification always comes with a set of challenges, the two main ones being:

• Label Imbalance: Some labels might appear very frequently in the dataset, while others are rare. Usually the more imbalanced is the data, the more difficult will be the model training.
• Correlation Among Labels: The inter-dependency between labels can complicate the modeling process, as the predictive model must effectively learn these dependencies.

From pre-processing to model validation, this process can take a lot of time and resources.

Luckily, we live in a world where pre-trained specialized language models are readily available. Additionally, providers offer access to large generalist models trained on massive amounts of text, known as Large Language Models (LLMs). Large Language Models are ideal for text analysis due to their ability to understand and generate human-like text. They can efficiently handle a wide range of tasks such as summarization, sentiment analysis and language translation in most languages. Their training on vast datasets allows them to recognize patterns and nuances in text, making them versatile and powerful tools for extracting insights and automating language-related tasks.

Why should you choose an LLM for text analysis ?

• Easy to use: LLMs are ready to use and do not require in-depth coding skills or extensive knowledge of data science and machine learning. Providers offer user-friendly APIs, making it simple to integrate LLMs into your applications, regardless of the programming language you use. Despite being a relatively recent technology in the mainstream, comprehensive documentation, tutorials, and community support are available to help users get started quickly. In most cases, the text does not even have to be preprocessed.
• Cost Considerations: Deploying a language model can be cost-intensive, especially if you don't already have access to a GPU. Nowadays, the best models for any language-related task have a neural network architecture. Using GPUs for running language models in production ensures efficient operation, delivering high performance and quick responses. However, setting up a GPU-equipped instance, whether in the cloud or on-premises, can significantly increase deployment and exploitation costs. For performing tasks on small text spans, considering the use of LLM providers' inference services can be more cost-effective because they charge per token (a sub-portion of a word).
• Performance: LLMs perform very well in most cases. Interestingly, the output of the model can be adjusted easily by providing it with one or more examples which will increase the quality of predictions.

It's often mentioned that LLMs are not deterministic, meaning the same query can produce different responses. While this is true, for simple tasks like classification, it is very unlikely to get different answers. Additionally, providers like OpenAI offer a seed parameter to achieve mostly deterministic results by imposing the random factor value.

Choosing to use an LLM over traditional NLP techniques can significantly speed up the development process for any text analysis task.

However, you will still need a test dataset to evaluate how well your solution performs in your specific use case. For multi-label classification tasks, a good testing dataset should reflect the complexity of your use case. Each sample should follow a consistent format, and the dataset should have a balanced distribution of label combinations, including samples with no label. If you know the real-world distribution of the labels, aim to match your test dataset to it. Generally, your test dataset should contain at least a few hundred samples, but the more labels you have, the larger your dataset needs to be. Ideally, the more the better.

Regularly generating test cases with fresh data should be a standard practice, as this enables you to monitor the model's performance in real-world conditions using real-life data.

Production challenges

Quality of Predictions and Monitoring

Classification is a crucial aspect of data analysis and strategic decision-making, demanding high-quality predictions to be effective. To evaluate the quality of predictions, there are numerous metrics available, each focusing on different aspects of performance (see annexe on classification metrics). For example, a metric such as Precision Score helps determine how well the classifier performs at prioritizing true positives over false negatives.

In production environments, tracking the outcome of each model call is vital. Storing the input and output traces enables post-analysis, helping identify accurate and inaccurate predictions. This trace analysis helps in developing strategies to enhance prediction quality and provides insights into token usage and call duration. To visualize the stored traces, you could use tools like LangSmith or Phoenix which provide tools to monitor LLM applications. Note that longer token sequences can lead to increased generation time, which should be considered when optimizing performance. Continuous evaluation of the model's performance with fresh data is also essential. Regular assessments ensure that the model remains relevant and accurate over time. This can be done through manual reviews or by employing another LLM to evaluate the classifier's output, a strategy known as LLM as a judge. This method helps in maintaining the model's efficacy by integrating real-world feedback.

Additionally, refining the model involves experimenting with different prompts to eventually improve results. When testing various prompts, it's important to store each prompt template along with its corresponding classification report. This allows for a thorough comparison based on chosen metrics to determine which prompt yields the best results.

An example of multi-label classification Report

import numpy as np
import pandas as pd

from sklearn.metrics import accuracy_score, classification_report, hamming_loss, zero_one_loss, jaccard_score, precision_recall_fscore_support, multilabel_confusion_matrix
from sklearn.preprocessing import MultiLabelBinarizer

def get_scores(labels_true: pd.Series, labels_pred: pd.Series, labels: list[str] = [], verbose=False) -> dict[float | dict[float]]: 
    mlb = MultiLabelBinarizer()
    if len(labels) == 0: 
        labels_true_binary = mlb.fit_transform(labels_true)
    else: 
        _ = mlb.fit([labels])
        labels_true_binary = mlb.transform(labels_true)

    labels_pred_binary = mlb.transform(labels_pred)

    subset_accuracy_score = accuracy_score(labels_true_binary, labels_pred_binary)
    ham_loss = hamming_loss(labels_true_binary, labels_pred_binary)
    multilabel_zero_one_loss = zero_one_loss(labels_true_binary, labels_pred_binary)

    report = classification_report(
        labels_true_binary, labels_pred_binary, 
        zero_division=np.nan, 
        output_dict=True, 
        target_names=mlb.classes_
    )

    jaccard_micro = jaccard_score(labels_true_binary, labels_pred_binary, average='micro', zero_division="warn")
    jaccard_macro = jaccard_score(labels_true_binary, labels_pred_binary, average='macro', zero_division="warn")
    jaccard_weighted = jaccard_score(labels_true_binary, labels_pred_binary, average='weighted', zero_division="warn")
    jaccard_samples = jaccard_score(labels_true_binary, labels_pred_binary, average='samples', zero_division="warn")

    label_wise_jaccard = jaccard_score(labels_true_binary, labels_pred_binary, average=None, zero_division="warn")

    report['weighted avg'].update({'jaccard score': jaccard_weighted})
    report['micro avg'].update({'jaccard score': jaccard_micro})
    report['macro avg'].update({'jaccard score': jaccard_macro})
    report['samples avg'].update({'jaccard score': jaccard_samples})

    for i, label in enumerate(mlb.classes_):
        report[label].update({'jaccard score': label_wise_jaccard[i]})

    if verbose: 
        print(f"""
Accuracy Score: { subset_accuracy_score }
Zero One Loss: { multilabel_zero_one_loss }
Hamming Loss: { ham_loss }
# =================================================================== #
{ pd.DataFrame(report).T }
# =================================================================== #
        """)

    return {
        'accuracy_score': subset_accuracy_score, 
        'zero_one_loss': multilabel_zero_one_loss,
        'hamming_loss': ham_loss,
        'report': report
     }

MultiLabelBinarizer() is used to convert the multi-label data into a binary format suitable for evaluation metrics.

Several metrics are used to evaluate the model's performance, see definition in annexe. classification_report computes precision, recall, f1-score and support at the same time, for each label and in average.

If this function is called in verbose mode, it displays a comprehensive multi label classification report. It always return the report structured as a dictionary.

By systematically evaluating performance, experimenting with prompts, and analyzing outcomes, you can continuously improve the classification process and ensure its effectiveness in real-world applications.

Price, Time, Quality trade-off

Since LLMs entered the mainstream, various techniques have been developed to address specific problems and improve prediction quality. Notable methods include few-shot prompting, chain of thought, and tree of thoughts, with new techniques emerging frequently. Each method uses different numbers of tokens in the input and output, impacting performance, speed, and applicability. Providers typically allocate 1/4 of costs to input tokens and 3/4 to output tokens. Understanding token distribution is crucial for managing costs and latency: it is essential to select the appropriate technique based on your budget and requirements, considering that more tokens lead to higher costs and longer execution times.

Techniques like chain of thought and tree of thoughts encourage the LLM to "think" more deeply. While these methods excel in performance for simple and complex problems, they are also more costly and time-consuming due to increased token usage. Chain of thought involves the LLM generating a step-by-step explanation of the final result. Tree of thoughts involves multiple "specialists" debating and presenting arguments to reach a final answer. Both techniques generate a substantial number of output tokens, leading to higher costs.

Example of Chain of Thoughts

I went to the market and bought 10 apples. I gave 2 apples to the neighbor and 2 to the repairman. I then went and bought 5 more apples and ate 1. How many apples did I remain with?
Let's think step by step.

Example of Tree of Thoughts

Imagine three different experts are answering this question.
All experts will write down 1 step of their thinking,
then share it with the group.
Then all experts will go on to the next step, etc.
If any expert realizes they're wrong at any point then they leave.
The question is: If I bought 10 apples, gave 2 to the neighbor and 2 to the repairman, then bought 5 more apples and ate 1, how many apples do I have left?

Few-shot prompting leverages the model's ability to generalize from limited examples (shots) and perform tasks with minimal explicit instruction. This technique is particularly useful in scenarios where acquiring large labeled datasets is impractical. It is versatile, adapting to a wide range of tasks and is easy to implement: you simply provide a brief explanation of the task along with a small set of input-output pairs as examples. One of the most challenging part is selecting appropriate examples because they are crucial for the model to accurately infer the task's structure and desired output format.

Exemple of Few-Shot Prompting (here it is a 1-shot example)

You are a 3rd grade student tasked with answering math problems in 1 line.
User: Lily has 12 apples. She wants to give an equal number of apples to her 3 friends. How many apples will each friend get
Assistant: 12 ÷ 3 = 4
User: If I bought 10 apples, gave 2 to the neighbor and 2 to the repairman, then bought 5 more apples and ate 1, how many apples do I have left?

For multi-label classification tasks, the minimal required output is a list of predicted labels. Few-shot prompting may involve a larger input token volume when incorporating numerous examples, but ultimately results in fewer tokens in the output by ensuring a concise and easily exploitable format. By focusing on a controlled output format, this approach offers an economical solution, as four words in the input cost the same as one word in the output.

For scenarios where budget and latency are not a problem Chain of Thoughts or Tree of Thoughts coupled with a well structured prompt and answer can also be a viable alternative.

Rate Limiting for API Calls

In scenarios where APIs impose limits on the number of requests that can be made within a specific timeframe, implementing a rate limiter becomes crucial. This ensures that applications stay within acceptable usage boundaries and prevent overwhelming the API provider with excessive requests. The ratelimit package simplifies this process by providing decorators that seamlessly integrate rate limiting into your Python functions.

Lets break down how it works:

from openai import OpenAI
from ratelimit import limits, sleep_and_retry

RATE_LIMIT_PERIOD = 60  # 60 seconds (1 minute)
RATE_LIMIT_CALLS = 50   # 50 calls per minute

def get_client():
    client = OpenAI(api_key=API_KEY)
    return client

@sleep_and_retry
@limits(calls=RATE_LIMIT_CALLS, period=RATE_LIMIT_PERIOD)
def get_chat_completion(messages):
    client = get_client()
    chat_completion = client.chat.completions.create(
        model=MODEL_NAME,
        messages=messages,
        stream=False,
        temperature=0,
        seed=424242
    )
    return chat_completion

RATE_LIMIT_PERIOD specifies the duration of the rate limiting window in seconds.

RATE_LIMIT_CALLS indicates the maximum number of API calls allowed within the defined period.

@limits(calls=RATE_LIMIT_CALLS, period=RATE_LIMIT_PERIOD) decorator enforces the rate limit on the get_chat_completion function.

@sleep_and_retry decorator wraps around the rate-limited function. When the rate limit is exceeded, it handles the excess calls by delaying and retrying them once the current rate-limiting window expires.

In our example, get_chat_completion is a function that interacts with OpenAI's chat API (client.chat.completions.create). If the function is called more than 50 times within a minute, the sleep_and_retry decorator ensures that the excess calls are delayed until the rate-limiting period resets.

By adhering to rate limits, you can ensure that your application interacts responsibly with APIs, avoiding service interruptions and potential outages.

A Multi-label Classifier with Few-Shot Prompting and OpenAI API

The following example presents how to make a multi-label classifier that tags short pieces of text. The classifier can select zero to many categories from 'sports', 'politics', 'economy', and 'entertainment':

def intersection(lst1, lst2):
    return list(set(lst1) & set(lst2))

def get_labels_from_chat_completion(
    chat_completion, 
    labels_wanted: list[str] = None, 
    labels_prefix: str = None
) -> list[str]: 
    
    if len(chat_completion.choices) == 0 :
        return []
    
    chat_completion_message_content = chat_completion.choices[0].message.content

    if labels_prefix is not None: 
        chat_completion_message_content = chat_completion_message_content.replace(labels_prefix, "")

    labels_str = chat_completion_message_content.strip()
    
    if len(labels_str) == 0: 
        return []

    labels = labels_str.split(',')
    cleaned_labels = [label.strip() for label in labels]

    if labels_wanted is not None: 
        cleaned_labels = intersection(labels_wanted, cleaned_labels)

    cleaned_labels.sort()
    return cleaned_labels 

def multi_label_llm(content_txt: str): 
    if len(content_txt) == 0:
        return []

    messages = [
            {
                "role": "system",
                "content": "You are a multi-label classifier tasked with identifying appropriate tags for a given text. Return the tags as a simple list separated by commas. If no tags apply, return nothing. The available tags are: 'sports', 'politics', 'economy', and 'entertainment'. Always consider the context of the text to determine the relevance of each tag."
            },
            {
                "role": "user",
                "content": "Text: 'Election debate highlights key economic policies.'"
            },
            {
                "role": "assistant",
                "content": "Tags:politics,economy"
            },
            {
                "role": "user",
                "content": "Text: 'AI will take on the world.'"
            },
            {
                "role": "assistant",
                "content": "Tags:"
            }
    ]
    messages.append(
        {
            "role":"user",
            "content": "Text: '{content_txt}'"
        }
    )
    chat_completion = get_chat_completion(messages)
    labels = get_labels_from_chat_completion(chat_completion, labels_wanted=['sports', 'politics', 'economy', 'entertainment'], labels_prefix='Tags:')
    return labels
    
multi_label_llm(
    "Stock market hits new highs amid economic boom."
)

intersection compute the intersection of two lists, i.e. the elements that are common to both lists.

get_labels_from_chat_completion processes the chat completion response to extract and clean labels. It handles a possible response prefix, splits the labels from the text into an array, trims whitespace, filters out unwanted labels and sorts the final list of labels.

multi_label_llm processes a text through a LLM that predicts which labels apply to the text. The context and goal of the task are explained in the system prompt. Two pairs of examples are added to the messages stack to help the model understand the format of the answer and the subtleties of the task. The text we want to process is provided as the last element of the messages stack, which is then sent to the LLM. Finally, the answer is formatted to return an array of strings.

Note that with Few-Shot Prompting, adding more labels to the classification task can reduce precision. The language model might struggle to accurately identify the most relevant labels when more options are available, potentially leading to less precise tagging. If this is the case and your labels allow it, a good workaround is to perform consecutive calls: first, a general one to determine broad categories, which can then trigger more specialized labeling calls. It's also worth experimenting with more complex techniques that are less affected by this issue.

Conclusion

Leveraging Large Language Models (LLMs) for Multi-Label Classification in text analysis presents a powerful alternative to traditional Natural Language Processing (NLP) techniques.

The capabilities of LLMs to handle complex and nuanced data make them highly suitable for tasks where multiple labels are needed. With their user-friendly APIs, LLMs offer ease of use, cost-efficiency, and high performance, enabling rapid deployment and effective handling of real-world data challenges.

By incorporating advanced techniques like few-shot prompting, chain of thought, or tree of thoughts, users can further enhance the precision and applicability of these models, ensuring robust and scalable solutions for diverse text analysis needs.

Regular evaluation and prompt refinement are key to maintaining the accuracy and relevance of the model in production environments, ultimately driving better decision-making and strategic insights.

Annexe: Multi Label Classification Metrics Overview

Subset Accuracy score

The Subset Accuracy Score, also known as the “exact match ratio”, is the strictest measure. It requires that each set of predicted labels for a sample corresponds exactly to the set of actual labels.

The Accuracy is computed for every sample then averaged on the whole set. The more the better. Ranging from 0 to 1.

Subset Zero One Loss

Zero One Loss is similar to subset accuracy, but rather than being a score, it's a loss where 0 means no error and 1 means at least one error in the prediction.

The Loss is then averaged on the whole set. The less the better. Ranging from 0 to 1.

Hamming Loss

Hamming Loss measures the fraction of labels that are incorrectly predicted, i.e. the fraction of times the wrong label is predicted, independently of the other labels.

The Loss is then averaged on the whole set. The less the better. Ranging from 0 to 1.

Precision

Precision measures the proportion of positive identifications that were actually correct. It is the score to prioritize if we want to avoid false positives: the precision is intuitively the ability of the classifier not to label as positive a sample that is negative.

For multi-label classification, multiple strategies for averaging are available:

• micro: Calculate metric globally by counting the total true positives, false negatives and false positives.
• macro: Calculate metric for each label, and find their unweighted mean. This does not take label imbalance into account.
• weighted: Calculate metric for each label, and find their average weighted by support (the number of true instances for each label). This does handle well label imbalance
• samples: Calculate metric for each sample, and find their unweighted mean. In binary or multiclass setting: this strategy is equivalent to the accuracy score, but not in multilabel settings where the classifier allow samples to have multiple labels. This strategy is valuable if we want to understand how well the model captures the correct labels for each data point.

The more the better. Precision is ranging from 0 to 1.

Recall

Recall measures the proportion of real positives that have been correctly identified. It is the score to prioritize if we want to avoid false negatives: the recall is intuitively the ability of the classifier to find all the positive samples.

For multi-label classification, the same multiple strategies as Precisions are available for averaging (micro, macro, weighted, samples).

The more the better. Recall is ranging from 0 to 1.

F1-score

F1-score is the harmonic mean of Precision and Recall. It combines both metrics. The relative contribution of precision and recall to the F1 score are equal: it is the score to prioritize if we want to avoid both false positives and false negatives.

For multi-label classification, the same multiple strategies as Precisions are available for averaging (micro, macro, weighted, samples).

The more the better. F1-Score is ranging from 0 to 1.

Jaccard Score

The Jaccard Score (or Index) measures the similarity between two sets of labels, defined as the size of the intersection divided by the size of the union of the sets of labels.

For multi-label classification, the same multiple strategies as Precisions are available for averaging (micro, macro, weighted, samples).

The more the better. Jaccard Score is ranging from 0 to 1.

Sources

https://en.wikipedia.org/wiki/Multi-label_classification

https://scikit-learn.org/stable/modules/model_evaluation.html#multilabel-ranking-metrics

https://en.wikipedia.org/wiki/Jaccard_index

https://scikit-learn.org/stable/modules/multiclass.html#multilabel-classification

https://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics

https://platform.openai.com/docs/guides/text-generation/chat-completions-api

https://platform.openai.com/docs/guides/text-generation/reproducible-outputs

https://platform.openai.com/docs/api-reference/chat/create#chat-create-seed

https://www.promptingguide.ai/techniques

When it comes to deal with historical data, it is a common task to work with the most recent and fresh data to perform analysis: that's where the ranking comes in.

The rank of an object is defined as a relationship between a set of items such that, for any pair of items, the first is either "ranked higher than", "ranked lower than", or "ranked equal to" the second.

Databricks implementations

It is not always possible and sometimes does not make sense to assign rankings uniquely. Depending of the use case, some strategies might be better than other. A common shorthand way to distinguish the best ranking strategy for your use-case is by the ranking 4 items:

• The first item ranked ahead
• The second and third, which compare equal (ties)
• The fourth item ranked last

The object of the different methods is to define the value of the ranking for equal items and items around them.

Databricks implements 3 ranking strategies.

Standard competition ranking “1224”

This ranking strategy is frequently adopted for competitions, as items that are equal receive the same ranking number, and the following item rank is its position in the ordered list.

In the competition example, it means that if two (or more) competitors tie for a position in the ranking, the position of all those ranked below them is unaffected.

Databricks implements this ranking with rank() over subsets of data called partitions.

SELECT 
    id_runner
  , rank() OVER(PARTITION BY race ORDER BY time_to_finish DESC)
FROM competitions

According to the documentation, rank will produce gaps in the ranking sequence but will not break ties.

Dense ranking “1223”

In dense ranking, items that compare equally receive the same ranking number, and the next items receive the immediately following ranking number.

Databricks implements this ranking with dense_rank() over partitions.

SELECT 
    id_runner
  , dense_rank() OVER(PARTITION BY race ORDER BY time_to_finish DESC)
FROM competitions

According to the documentation, dense_rank will not produce gaps in the ranking sequence and will not break ties.

Ordinal ranking “1234”

This strategy it meant for scenarios where the uniqueness of the rank is required.

In ordinal ranking, all items receive distinct ordinal numbers, including items that compare equal. The assignment of distinct ordinal numbers to items that compare equal can be done at random, or arbitrarily, but it is generally preferable to use a system that is arbitrary but consistent, as this gives stable results if the ranking is done multiple times.

Databricks implements this ranking with row_number() over partitions.

SELECT 
    id_runner
  , row_number() OVER(PARTITION BY race ORDER BY time_to_finish DESC)
FROM competitions

According to the documentation, row_number will not produce gaps in the ranking sequence but will break ties.

If the order is not unique, the result is non-deterministic.

An example of an arbitrary but consistent system would be to incorporate other attributes into the ranking order

Comparing methods

Now let's wrap up everything and compare the methods. It's up to you to use the one that best fits your needs!

SELECT 
    id_runner
  , race
  , rank() OVER(PARTITION BY race ORDER BY time_to_finish DESC)
  , dense_rank() OVER(PARTITION BY race ORDER BY time_to_finish DESC)
  , row_number() OVER(PARTITION BY race ORDER BY time_to_finish DESC)
FROM competitions

id_runner | race | rank | dense_rank | row_number |
----------|------|------|------------|------------|
        21|     A|     1|           1|           1|
        56|     A|     2|           2|           2|
        87|     A|     2|           2|           3|
        18|     A|     4|           3|           4|
----------|------|------|------------|------------|
        87|     B|     1|           1|           1|
       ...|   ...|   ...|         ...|         ...|

Sources

Ranking - Wikipedia

Wikimedia Foundation, Inc.Contributors to Wikimedia projects

rank ranking window function

Learn the syntax of the rank function of the SQL language in Databricks SQL and Databricks Runtime.

Databricks

dense_rank ranking window function

Learn the syntax of the dense_rank function of the SQL language in Databricks SQL and Databricks Runtime.

Databricks

row_number ranking window function

Learn the syntax of the row_number function of the SQL language in Databricks SQL and Databricks Runtime.

Databricks

Metabase is a great tool that offers a lot of functionalities, but sometimes I would like to get even more out of the tools I use. In particular, I am talking about filters. While filters are a valuable and essential functionality to perform dynamic analysis, they present some limitations and can lead to confusing situations, diminishing the end user experience. Below, I show a way to tweak and customize 2 types of filters in Metabase to make them perfectly match your user needs!

Limiting the number of values in drop down filters: 

In this case, I want to display the sales which have been made in a subset of countries (let’s say 10 countries). In the database there is a table listing every country where we sell products. I would like to use this table to build a dropdown filter for my question and show only countries from the subset I'm interested in, instead of including the whole list. With the method above, the dropdown list will list all countries available from the database. I don’t want to scroll to the bottom of the list or type the name every time I want to select France or the UK for example. I want to show only the 10 countries my users are interested in. A solution is to use the available custom dropdown list filter but you have to type the whole list for each new question which alright for a very small list but  nonsensical in the world of automation and hence not the optimal solution.

The most efficient way to use a dropdown filter in Metabase is to make it of the type field filter and map it to the field (in this case country) of the table the data is stored in. Dropdown is not the only categorical filter available in Metabase. Sometimes it’s preferable to define a simpler filter, like a string search field or a custom list of strings from a text filter, although multi selection is not available for this kind of filter.

To solve this problem, we use a dedicated question that lists the countries we are interested in (which we use as a location dimension for other questions). 

There is an easy way to do this in Metabase:

1. As usual, we start by making the filter a field filter.

2. You will see that the dropdown list contains all the countries that are included in the table. To narrow the available choices, create a new question (or native SQL depending on your requirements) which shows only the list of countries that you want in the dropdown list. Save it in the same place where you will save the main question.
3. Click on the Edit button that you see on the right side of Dropdown list in the image above. Then click on From another model or question and select the question you just created which has the list of countries. Select the field (name in my case). Click on Done.

4. Et voilà, when I click on the filter, I see only the 10 countries that I’ve listed. And I can multi-select!

Adapting normal text filters to handle multiple comma-separated values:

Here the data is user created, meaning it does not already exist in this form in the database, so we cannot directly map the filter to a field in a table and make it a dropdown list with multi selection possible (as explained in the previous part).

This issue can be solved with the FIND_IN_SET function from MySQL 8. The way to make this possible is to use a text filter in which users can  type the values they are looking for separated by a comma. Let’s say we analyze the sales of the company grouped by the departments in France. We want to allow the user to type, e.g. 11,12,13,14,15 and have the question show the results for these 5 departments. We had to find a way to retrieve each of the data that belongs to one of these departments, which from a technical perspective is the same as looking for the department of each line of the sales table in the set of user provided departments. 

In France, most of the department numbers are defined as the first 2 digits of the postal / zip code (5 digits). In the following I will follow this rule for the sake of simplicity.

Let’s define:

👉 postcode is the field containing the zip code of the customer buying a product from our shop

👉 department is the custom field derived from postcode containing the department of the sale

• With MySQL 8: LEFT(postcode, 2) AS department 

👉 {{departments}} is the filter of type text (input box format) that can receive multiple department numbers separated by a comma

The function FIND_IN_SET of MySQL 8 takes two parameters and returns the position of the 1st parameter (the string to find) in the 2nd parameter (a set of comma separated strings).  If the 2nd parameter doesn’t contain the first parameter, the function returns 0.

The idea is to find the value of the department  in {{departments}}. If it is not found, FIND_IN_SET will return 0. Thus, I just had to slice the results of my query with the following condition:  

FIND_IN_SET(LEFT(postcode, 2), {{departments}}) > 0

As we can see below, we get exactly the 3 departments typed in the filter. There is no limit to the number of departments we can type in one go.

A limitation, however, is that the string in which the search is performed has to be comma separated. If for some reason, the users want another type of string delimiter for the filter, FIND_IN_SET will not work. You could use REGEXP or declinations of this function to solve this problem.

Another limitation of this technique is that the user has to know all the classes of the data set in order to access full information: the classes are not listed in a dropdown list and the users can forget one in their analysis.  

And that is how we can tweak these 2 types of filters in Metabase to make them show us what we want. It’s a little bit of work, but it works!

References:

https://www.metabase.com/docs/latest/questions/native-editor/sql-parameters#field-filter-syntax

https://www.metabase.com/learn/sql-questions/field-filters

https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_find-in-set

https://dev.mysql.com/doc/refman/8.0/en/regexp.html

Firebase is a BaaS (Backend as a Service) platform for building mobile and web applications. It was founded in 2011 by Andrew Lee and James Tamplin as a platform to help developers build web and mobile applications without worrying about infrastructure. Initially, the platform offered a real-time database, authentication, and hosting features. Google then acquired the platform in 2014, which helped its growth and development, with the continued expansion of its range of services. It is a simple and user-friendly platform that doesn’t require complex configuration. 

Some of the features of firebase are

• Real-time database, 
• Firestore,
• Authentication,
• Cloud storage,
• Cloud Messaging
• Cloud Functions, 
• Crash Reporting,
• Analytics,
• Hosting.

Realtime Database: It is a NoSQL cloud-hosted database that allows to store data and sync across multiple clients or devices in real-time.

Firestore: It is an advanced NoSQL cloud-hosted database with real-time synchronization, powerful querying capabilities, and offline support.

Authentication: It provides a simple and secure way to authenticate users with 

• email and password, 
• social media logins,
• custom authentication.

Cloud Storage: It provides scalable and secure cloud storage for files including media files.

Cloud Messaging: It allows sending notifications, data messages, and messages with both notification and data payloads to users on different platforms.

Cloud Functions: It allows developers to write serverless functions in JavaScript that can be triggered by events in your app or other Firebase services.

Hosting: It provides fast and secure static hosting for your web app, with automatic SSL encryption and CDN integration.

Analytics: It provides insights into user interactions with your app, including behavior, demographics, and more.

Crash Reporting: Firebase's Crash Reporting service helps developers identify and fix crashes and errors quickly and easily.

Pricing

Firebase offers a range of pricing options, with a free tier for those getting started with the platform. Paid plans are available for those who require additional storage, bandwidth, or features.

Free tier Real-time database limitations

• 100 Simultaneous connections
• 1 GB of storage
• 10 GB/month of data download
• No multiple databases per project

Drawbacks

While the platform offers numerous benefits, it does have some drawbacks to consider. 

Limited Querying: The real-time db doesn’t support complex querying like SQL databases. For complex querying, additional data processing is required on the client side. This can be overcome by using Firestore which provides advanced query handling. 

Vendor Lock-In: Locked into using Google’s services. Hard to migrate to another platform which requires a lot of work.

Cost: It can be expensive for Large applications and high-traffic websites. 

Firebase Real-time DB vs other NoSQL DBs

Let’s compare the real-time NoSQL db provided by Firebase with other NoSQL dbs like Amazon DynamoDB, Cassandra, Couchbase, and MongoDB based on the following parameters. 

Data Model: The way data is structured, stored, and accessed 

Real-time Synchronization: The ability to update data across multiple clients in real-time.

Data Scalability: The ability to handle increasing amounts of data 

Data Querying: Process of retrieving the data from db

Hosting & Management: The way the db is made available and managed.

How we utilize Firebase 

We utilize Firebase in our organization in the following ways.

Authentication: To authenticate users in one of our applications using email and password.

Notifications: To send notifications to users in our application across multiple platforms 

Firebase Queue: To manage events between micro-services

Maintenance: To manage the application. With the change of a property, the application can be put in maintenance mode. 

Feature Flags: To enable or disable a specific functionality of an application. 

Force refresh App: To force refresh the application for all active users at a time. 

Overview

Firebase is an excellent platform for quickly and easily building high-quality, scalable mobile and web applications. With its range of powerful features and easy-to-use tools, it is a popular choice for many around the world including us. 

Alternative BaaS platforms

If Firebase hasn’t impressed you yet, there are numerous BaaS platforms out there you can check out. Some of them are,

AWS Amplify: Amazon's BaaS platform that offers data storage, user authentication, and more.

Backendless: A BaaS platform that offers data storage, user authentication, and serverless functions.

Kinvey: A BaaS platform that offers data storage, user authentication, and serverless functions, as well as integrations with various third-party services.

Parse: An open-source BaaS platform that offers data storage, user authentication, and serverless functions.

Business context

Business context has two components:

1. The product vision and strategy

The product vision describes the future we are trying to create, typically somewhere between two and five years out. The product vision is one of our most effective recruiting tools, and it serves to motivate the people on your teams to come to work every day.

The product strategy is our sequence of products or releases we plan to deliver on the path to realizing the product vision.

Product vision and strategy describe the big picture of what the organization as a whole is trying to accomplish and the plan for achieving that vision.

2. The business objective

This describes the specific, prioritized business objectives for (each) product team. Tell the team what you need them to accomplish and how the results will be measured, and let them figure out the best way to solve the problems.

It is the management’s responsibility to provide each product team with the specific business objectives they need to tackle. The difference is that the product team is now prioritizing business results, rather than product ideas.

The advantage of this way of working is that the teams are much more motivated when they are free to solve the problem the best way they see fit. Teams are designed to be in the best position to solve these problems and it is all about outcome rather than output.

Product Strategy

To illustrate what is defined by Management and what is defined by the Product take a look at the Product Strategy Stack in the image below. Tops down from mission to product goals all the effort is on correctly defining, planning and alignment on the execution plan; creating any good products or leading any company well without having a clearly defined mission and how to reach these objectives i.e. company and product strategy is not possible. Bottoms up is evaluation; communicate the status of execution,  the relatedness and success of the product by tracking how well the product is reaching defined organization’s business objectives.

Source: Reforge

Building on Product Strategy Stack, an example of Product Strategy / Business Objectives and how it will be translated into Product is explained in the following. This illustrates the definition and evaluation importance. Airbnb had the mission to serve the people who want to rent mattresses in someone else’s house. Airbnb took the strategy of showing pictures of the rental places to attract more views and conversions. So the product enabled the feature and it was tracked initially, with an alarming behavior! Pictures were too poor in quality when uploaded by owners, so to meet the objective of conversions, founders hired professional photographers to take high-quality pictures to put on the website. The result was that the product feature was appreciated better and conversions increased exponentially for Airbnb.

Product Development Hierarchy

Getting it more practical and zooming into the Goals and how they are broken down to User stories, the image below illustrates the Product Development Hierarchy with an example of an online streaming platform. Here, a possible goal in the context of the product is illustrated.

Goals are what you want to achieve to meet your short term or long term vision and Initiatives are the high-level efforts that will help you achieve your goals. Success of an initiative, depending on the context of it can be measured by: objectives of initiative being met and the benefits are measurable, quality is met or exceeded the expectation, customers are satisfied, the budget for completion is well respected and support for initiative is well planned.

Alignment of Product Vision, Product Strategy and Business Objectives

Marty Cagan emphasizes in his book, INSPIRED, the importance of aligning product vision, product strategy, and business objectives in order to create products that customers love. He argues that without a clear product vision, product strategy, and business objectives, companies will lack the focus needed to create products that truly meet the needs of their customers.

Cagan emphasizes the need to ensure that the product vision, product strategy, and business objectives are closely linked in order to ensure that the product being created aligns with the company's overall goals and objectives. Cagan argues that a successful product requires a clear understanding of the customer needs and an effective product strategy that is tailored to meet those needs.

Conclusion

To conclude, for a product team to be empowered and act with a meaningful degree of autonomy:

• The team must have a deep understanding of the broader context. [product vision to inspire and set direction, and product strategy to get there];
• Clarity from leadership on the critical pieces of context; lack of clarity raises problems and leads to ambiguity over what a team can decide and what they can’t.
• Tell the team what you need them to accomplish and how the results will be measured, and let them figure out the best way to solve the problems.

Resource: INSPIRED: How to Create Tech Products Customers Love by Marty Cagan

Working with multiple people in a start-up project requires good communication and some conventions to accept to scale successfully and match the product need in productivity. However, because almost infinite possibilities exist to code the same feature, we need to get our stories straight and create guidelines that are easy to follow, which benefits the team's efficiency but also helps in delivering high-quality features, easy maintenance, and good readability.

Our vision

According to the need, we started to create the core principles that we believe will make important improvements in the quality of our deliverables. We have 3 core principles that are very close to each other and will lead our thoughts on changing/adding new good practices.

Component’s independence

The first principle is that we should be able to create independent components that are - in theory - easy to extract from one project to another. This is the initial goal of the component system in a front-end framework like VueJs. We want to keep this ideology and use it across our daily code.

Creating components will help in code readability and also help us to factorize and avoid having code duplicates. Because having twice the same code means that if you want to change it, you have to change it everywhere, and also, the same if you want to remove it or improve it.

Despite the fact that a more independent component can be easily extracted, it also means that we reduce the things that will impact our own components, which makes it easier to maintain. This does not mean that we don’t create relations between our files, in fact, we do it a lot but we try to keep a simple communication, easy to understand, and easy to find between our component and external impacts like API, props inheritance, compositions, etc…

One example that would conflict with this vision would have been if we created an entire page in a single component. Data are being fetched from one place (Store, API call), then treated from another place. Admitting we are receiving the data (helpers or libraries) and then received in our component by inheritance (mixins). Hence, everything is magic and nothing tells you in this component that you have the data you want, formatted the way you want unless to walk through the whole path.

Smart & dumb components

The second pillar is smart and dumb components which can be a little bit different from what you can read about those terms elsewhere.

In our case, our dumbest components are when it has no business logic inside, its goal is to show some data he doesn’t even know about. Most of our UI components like buttons or modals are dumb components.

We call a component smarter when it adds strong context or business logic. Most of the time, it is built from dumber components assembled with context or specific data. For example, we can imagine a component that renders a card for a premium member with their name, and avatar, … So it is built from an avatar and a card (dumb) component and thanks to its context, creates a member card.

Doing that means that our dumbest components need a lot of effort in the making because they need to be very adaptive and robust to be used everywhere. The smarter the component is, the easier it is to create because it is just placing some already existing component and matching it with our context.

As you can see it’s not a two-sided concept, a component is not either dumb or smart. It is just smarter than one and dumber than another so we can think of that as a “smart scale”. A generic button is dumber than an “add to cart” shopping button, which is dumber than an order card with this button, which is dumber than a shopping cart page.

Black box design

The last principle wraps up the first 2. Even if dumb components might be complex to create, we still need to be easy to work with inside our application. To do that, we have commonly agreed that we don’t need to know how the component works inside, we just want to use it so we need the communication interface contract with it to be crystal clear and self-documents the component itself. After that, all devs should be able to use it freely and have what they expect.

Of course, this concept doesn’t work only for the dumbest components but for any components, and this is why we have a Storybook design system displaying our available components to make it much simpler to find and use the correct components.

Finally, we can imagine that if we need to work inside a dumb component which is a black box, it would be super hard to get in and we would need a front-end expert but due to the “Component’s independence” and “Smart & dumb design”, we believe it’s not that complicate to work inside and make improvements and we also added some extra guidelines to ease it a little more.

Matching Vue trends with our vision

In order to help the readability of our code, we have chosen to follow the Vue trends proposed by its community and adapted to the previous principles.

Strict Vue ESLint rules

At Jolimoi, we have decided to have a very strict technical environment and guidelines and we chose ESLint to handle most of the job for us. This way, most of the code is produced with the same conventions so we are more efficient in reading and understanding the code already in place.

So we decided to use Vue ESLint rules from their documentation and apply all the rules possible inside our code base. We applied all base, essential, strongly recommended, recommended, and uncategorized rules to our linter. Some of them were incompatible with our way of coding so we chose to disable them for now and we also configured some of the rules with the recommended settings most of the time. We are in the middle of a rewriting migration so we are adding these rules little by little in our codebase, but this is the goal we are aiming for.

Once it’s in place, and of course it breaks our CD/CI if we have any errors during the linting, all developers use it and agree with the code guidelines. For example, Vue gives us rules to forbid the creation of components of one word only. Every component must be at least 2 words combined in order to differentiate between HTML tags and Vue components, so it’s easier to see/read. We also have rules to disallow us to create anti-patterns like mutating props, or to forbid having a v-if at the same level as a v-for.

Composition API over Option API

Also, Vue appeals to us to use more composition API, especially with the script setup syntactic sugar. We wanted to use composition API because it adds much more possibilities to manipulate our code outside/across our Vue components which gets very important when you want to factorize your code and avoid duplicates.

Also, we believe that using composition is a good option to handle impacts coming from outside a component since it’s clear enough to understand where it comes from (and easy to navigate through it with a code editor) . Also, using composable for example, we can “scope” our impact and also get only the minimum required for our component to work properly, which is super compatible with our vision.

So we have decided to switch to composition API and script setup. In fact, it’s a simple conversion that even ChatGPT can do for us. On top of that, we have chosen a few guidelines to ease the skill improvement such as choosing ref() over reactive(). Ref() may be trickier than reactive() but works in all scenario and we believe that it helps us to understand better how Vue work. Also, we have decided to organize the most efficient way our code and group them: declarative variables (such as props, composable, emit), const/data, computed, methods, lifecycle methods, etc. so it’s easier to read and find what’s affecting what inside our component.

Also, since we are not in an object anymore, it feels obvious that having a huge component with a lot of code and complexity inside is getting very hard to read and maintain so we are more encouraged to extract code outside, especially with the full power of composition API.

For example, we have a feature flagging library that we want it to disable, regarding a tag which is a boolean get from Firebase, a specific part of our codebase. Previously we had a mixin given to every component in our application that initialized our firebase got all of the feature flags and then populate asynchronously some methods and data. It works well but we have a few problems:

• since it’s asynchronous, it’s difficult to evaluate whether the data is passed or not
• by looking at our component, it is impossible to know that we have these properties inside which makes it hard for newcomers to start working on the stack
• it’s impossible to use these feature flag checks outside a Vue component as it is today
• since we now want to use script setup, mixins are no longer compatible

So a better way to handle this would be to rewrite the mixin into a composable with something like const { isFeatureEnabled } = useFeatureFlag() so now, inside a component if we want to use it we import it so we know where it comes from and if we want to use it outside the component, we can.

Minimize store usage

We believe that using a global store like Vuex or Pinia is a core-breaking principle since it creates impact across all our components while nowadays, we can do almost everything we commonly do inside a Vue store with composition API to communicate across multiple components inside our application.

Most developers use Vue store in order to either store data we want to use across the entire application or store data in order to cache it. For the 2 needs, we have an answer: we use composition API and composable so we can use data across the app. For caching, the only thing we’d like to cache right now is API calls, and for that, Tanstack/query does the job for us perfectly.

For example, we like to use Vue store for authentication across the application and to have the user everywhere but with composition API, we can handle it super easily without any other libraries. We created a useAuth() composable that handles everything for us, which is Vue-reactive and can be easy either inside any components and also outside. Of course, it handles both sides: get the currently logged-in user but also some actions such as login or logout.

Conclusion

To conclude, all the guidelines you just read are a vision, it’s more of a goal to reach but it can happen that sometimes, for various reasons, we do “bad code” that doesn’t follow every guideline here. We want to involve developers to think of what they create and especially why they create it this way. Doing this helps us to decrease the chances of having struggles to maintain some code later.

We just believe that keeping our mind on this will help the product and the dev team to improve and it will ease our daily tasks.

At Jolimoi, the data assets play an important role in productivity, efficiency and visibility, that's why we decided to do a cleaning of the data visualization tool (Metabase). We divided the project into distinct phases due to the amount of assets in our  organization. Transforming from a noticeable disorder to a better structured tool which will positively  impact the company.

Our data visualization tool is a self-service business intelligence oriented that helps our teams to perform:

• Data exploration
• Create dashboards
• Generate alerts when data does not follow specific business defined rules

All of this will help stakeholders to speed-up and ease their decision-making and made everything much more transparent for the collaborators.

In this blog post, I will share the process and the lessons learned from the data team in this project.

Phase 1 - Interviews

This phase aims at understanding the importance of feedback from different teams using Metabase.

During the Interview Process, we interviewed our teams to understand how they interact with Metabase to identify pain points and inefficiencies in the current process.

Interview methodology

We selected key users from each team to participate in recorded  interviews. The goal was collecting insights and prioritizing future improvements.

Questions

Choosing the right questions was important for the interview process, we wanted to understand users roles  and how they interact with Metabase and the data in general. We defined 3 stages of questions:

• Own interaction stage where we want to understand how they process to get data in general terms we asked questions like: do you currently analyze any data? which? and how? How often? Do you have any periodicity usage on a daily, weekly, monthly basis?
• Metabase interaction stage where we ask about data visualization tools and how deep they know Metabase we asked questions like: Have you used any visualization tool before? Are you currently using Metabase? Do you know what metabase is?
• Role experience in Metabase stage to know how they incorporate it into their regular tasks and we asked questions like: Do you have some links or question IDs pinned or favorite to use questions saved that can be accessed easily? Is it easy for you to find the requests on Metabase by using the search bar?

Data collection

After conducting interviews, it was essential to prioritize improvements based on the data collected. This process aimed to address identified issues and concerns. By considering the feedback received during the interviews, we determined which improvements were most important and should be prioritized.

In this level we found some common issues like:

• 👉 Most of the users didn't  feel comfortable using metadata due to the amount of keywords and names found
• 👉 Users copy data in Google Sheets for their own analysis (high usage of raw data)
• 👉 Users are willing  to get more experienced in Metabase
• 👉 The most required improvement was to arrange all the questions and tags to  facilitate the search experience
• 👉 Users regularly use data from Metabase on a daily basis
• 👉 Some users knew how to use filters or summarize functions. This users use this features to search and refine data presented by Metabase
• 👉 Some users bookmarked their frequently used questions in the browser

Phase 2 - Cleaning Metabase old assets

As part of immediate actions we decided to go with bulk archiving questions based on different particularities scheduled for every quarter creating dashboards to track the cleaning process.

First we needed to identify assets that were not used for a long period of time and create confusion to users.

Cleaning assets rules

Data team must ensure this process is followed correctly and provide help to any other user when facing a problem for a missing request.

• 👉 Request with no views for the past 6 months or a maximum of 10 views, including all admin users requests that satisfy the conditions
• 👉 Request that had been archived for more than 6 months
• 👉 Every admin user will be in charge of cleaning his/her own assets periodically
• 👉 Rules will not affect personal collections of any user
• 👉 Requests created by new Data Team members or that are in Technical collection on Metabase will be excluded from automatic archival (this collection is used to stored some technical assets like Metabase models)
• 👉 We encouraged users to archive unused requests or move regularly used requests from their personal collections to a common collection
• 👉 Unused requests from users’ personal collection can be archived by the data team if it deems these requests not useful, without informing the user
• 👉 Metabase naming convention should be followed by all users for the requests that are moved to common collections
• 👉 Data Team is free to modify the request names according to the naming convention for the requests that will be moved from personal to common collections
• 👉 There are some assets that are only reviewed once a year, for these we have a special list to exclude them from archiving

Bulk cleaning

With the amount of different assets needed to be reduced, we test the cleaning process on a smaller scale before executing it in bulk.

When we do this process some points come up, we need to be careful in all the requests that appear in different scenarios, like requests that are consulted once a year or personal collections, admin requests that are a work in progress, we ensure the bulk cleaning did not affect the current operations of all the users.

Documentation updated

We create some referential documents regarding the process and rules to give the users a better understanding of the origins of the project and how we manage bulk erasing requests to maintain visibility to users.

Follow Up

We scheduled some slack alerts for this process to ensure the process is followed correctly and simultaneously.

We also organized Metabase training for basic and intermediate level to ensure the tool is well used and the rules well interpreted.

Conclusions

In conclusion, keep our Metabase clean is important for improving the user experience.

By regularly cleaning out these "old" assets, we can ensure that users are provided with accurate and up-to-date information, a cleaner interface simplifies the user journey, without being overwhelmed by unnecessary content and increase trust in the tool's reliability.

How we measure success?

At performance level, we create a "cleaning" monitoring dashboard. We used it to quickly track some KPIs regarding assets cleaned.

This displays key metrics, including the overall volume of archived requests as well as those archived within the last six months. This will provide us with valuable insights into the performance and effectiveness of the archiving process.

Simultaneously, we are enhancing the monitoring of active requests by implementing visualizations that help to identify requests that are receiving no views, allowing us to queue them for archiving once they meet the threshold.

Principe of Teleport

Teleport is a component that appeared in Vue 3.

This component allows displaying its content to slot elsewhere in the DOM. To do that the component has props “to”. This prop uses a CSS selector to know where to display the content of the Teleport.

With Teleport, it’s possible to interact on the DOM external of your component scope while using an ordinary component easily manipulated.

Teleport has another prop “disabled”. This prop allows disabling the teleport of the content and adding the content to the DOM normally.

Before Vue3, there existed another library: PortalVue. This library has the same concept and many developers from Vue community loved it and used it. Then the Vue Team was natively implemented on Vu3 with a new name: Teleport.

PortalVue still exists and has other features that don’t exist in Teleport.

Simple example

For a better understanding, here is a simple example. I

In a Vue 3 application, I create a simple component. In this component, I add a <div> with the text:

<template>
 <div>
   First Paragraph
 </div>
</template>

And in the <div> I add the Teleport component:

<template>
 <div>
   First Paragraph
   <Teleport to="#test"> Second Paragraph</Teleport>
 </div>
</template>

In my Teleport, I add text in the slot and a CSS tag in props. This CSS tag targets an element HTML with the test in id.

After that in the App.vue I add ExampleTeleport component:

<template>
 <div>
   Text in App.vue
   <ExampleTeleport></ExampleTeleport>
 </div>
</template>

This is the result for now:

And if I want to add <div> with the id:

<template>
 <div id="test"></div>
 <div>
   Text in App.vue
   <ExampleTeleport></ExampleTeleport>
 </div>
</template>

The content in Teleport is shown before the first paragraph:

And if you want to show the second paragraph after the first text is possible and very simple! Teleport has another prop: disabled. This prop disabled the teleport feature. For example, if I add disabled props on my component

<Teleport disabled to="#test"> Second Paragraph</Teleport>

Now the second paragraph is after the first paragraph.

Why use Teleport?

After discovering how to use the Teleport component, we will now see why to use this component and what its advantages are with a few examples.

Reason 1: Manage the DOM outside of a component scope

The first reason and that you need to manage the DOM outside of your component scope. With Teleport is possible to manage the props, data, method, etc. in your component and display the content elsewhere in the DOM.

It’s very useful to avoid many problems with CSS in particular with position, z-index, overflow, display, etc., or to don’t have a problem with the HTML semantics.

Example: Modal component

A good example of that is if you want to create a modal component. With this component, you can have many problems with CSS because we use position or overflow attributes and with the HTML semantics.

Let’s build a simple modal component:

<script setup lang="ts">
const props = defineProps<{open: boolean}>();
</script>

<template>
  <div v-if="open" class="modal">
    <slot></slot>
  </div>
</template>

Now if I add my Modal on the button:

<button>
  I'm a button !
  <Modal :open="true">
    <p>Waw ! It's an awesome modal !</p>
  </Modal>
</button>

We have two problems:

• The semantic HTML it’s not good and if I click on the modal, I click on the button
• The style of the button can have an impact on my modal

If I use Teleport I can resolve all problems with that.

The first step is to add a target element (on my index.html for example):

<div id="content-modal"></div>

After that, I can add Teleport to my Modal component:

<template>
 <Teleport to="#content-modal">
   <div v-if="open" class="modal">
     <slot></slot>
   </div>
 </Teleport>
</template>

So we have that on the result (I add style for my button and my modal):

If you try, you can see, we don’t have a problem with the HTML semantics.

Reason 2: Write a more readable code

The second reason to use a Teleport component is to have a readable code. Indeed, in some cases, Teleport allows simplifying the code.

Example: Alert component

Imagines an application where you want to display an alert message at the top of your page. The alerts can be created everywhere in your code (in the case of a form-sending error for example).

For that, you need to transport your data from one component to another component by using the store (Vuex, Pina, etc.) for example.

But with Teleport, you can have another solution. I created a simple alert component:

<template>
 <div class="alert">
   <slot></slot>
 </div>
</template>
<style scoped>
.alert {
 border-radius: 5px;
 color: white;
 background: #00bfff;
 padding: 10px;
}
</style>

And I add the teleport component:

template>
 <Teleport to="#content-alert">
   <div class="alert">
     <slot></slot>
   </div>
 </Teleport>
</template>

Now, if I add an alert to my project my alert is shown before the content:

 <div id="content-alert"></div>
 <div>
   I'm the content on the website
   <Alert>I'm an alert message !</Alert>
 </div>

The result is:

If I had not used the teleport feature then I needed to transport my data with a store for example and my code would have been more complicated.

In the case, you add a new alert, the second will insert in the DOM after the first:

<div id="content-alert"></div>
 <div>
   I'm the content on the website
   <Alert>I'm an alert message !</Alert>
   <Alert>I'm a second alert message !</Alert>
 </div>

Teleport is a very cool feature (opinionated)

As you can see Teleport is a very cool feature! He use easier our life with the management of the DOM outside our scope component.

It’s not the only component created with Vue3. With this version, we have some components with cool features to create our web applications. 😄

However, you need to do good use of this feature! By controlling the DOM outside to scope of your component if you abuse to use Teleport you can have side effects and it will come to hard to debug.

Moreover, this component has inconvenient because if we target an element inside on your Vue application and this element isn’t mounted, the Teleport doesn’t work. This problem is supposed to be solved this year. You can see more details on the roadmap video.

I always had an appetite for accompanying people in their technical projects. To me, the interaction we have with the technical tools is fascinating (much more than the technical tools themselves). So naturally, after strengthening some pure programming abilities, I decided to take on the “Engineering Manager” challenge. Not as a logical promotion after years of service, but as a turning point in my career. And this was a real decision to become an effective one. Learning and working my way up to the proper mindset and skill set.

One of the many books I devoured to improve my understanding of this job was “Taking People With You” by David Novak. And this one stood off!

So much so that I wanted to share what had an impact on my progression as a fresh manager.

#1 Being your best self

Ok, this one is probably not quite exclusive to the Engineering Manager profession. But being your best self, or the best version of yourself is particularly important when it comes to this job.

People will turn to you, either actively or passively. Just look at how you are, and what you do, and it will impact them. After all, you are a manager, and it makes you at least an implicit reference. So you truly need to embody your values, be as sharp as you can be, and live out every piece of wisdom you will ever want to share. Then you can understand what it means, what can be challenging, and where the struggles are while being a doable proof.

So this piece of advice helps to have a positive impact on the team.

#2 People Do Things Even They Didn’t Know They Could Do

This piece of advice resonates a lot with another key concept I hold dear which is “naive optimism”. We don’t necessarily plan to do “amazing things”, we plan to do regular things, and they turn out to be amazing. But for this to truly work, you have to shun all the possibilities it can go wrong, and not focus on the fact you cannot do it. Try and learn. You’ll probably fail, but if you succeed, what a victory! And interestingly, it is exactly how we learn to walk. We have no experience, no muscles, no muscle memory, nothing. On paper, it is a lost cause, and a robotic engineer will say it to you: walking is a marvelous feat. But we get there eventually! After a lot of falls, and more perseverance.

It is hard to have this naive optimism for ourselves. But if we do, and if we manage to help others, it can lead to interesting results, and drastically open up the possibilities.

#3 Being an avid learner

I considered myself pretty solid when it comes to learning and oh boy I was wrong. The way I saw learning was always very academic, structured, planned, and so on.

But here, knowledge is not acquired, it is devoured. There is a certain chaos in that process, which leaves room for unexpected opportunities. Everything is a source of learning, everything and all the time. It fires in all directions, on every topic. And instead of properly framing one topic and learning all its details, you have a learning trend about a concept. The trend brings you more connections, often outside the box (because you get rid of the frame).

And being an avid learner becomes more than a piece of advice, it’s a concept that shapes how you see the world and everything around you.
It is again not tied to this job and is globally handy (especially when you are starting).

#4 Recognition

This incentive straightforwardly echoed my fresh experience. I find it much easier to work around mistakes, failures, and problems. Easier to have the feedback, easier to spot, and easier to address. Whereas it feels tricky to find all the wins, as there are a lot of silent ones. But then the conveyed message is all twisted (focusing on negative things), and it could greatly impact the motivation of the team.

Getting to see the extra mile done here and there, managing to notice the further care given on tasks, and then simply giving some attention and praise about it. I think it is both trickier and more rewarding. Not only do you build a positive feedback loop, but you also reinforce a nice work atmosphere. Oh, and it does feel good for everyone to bring recognition to the table! Which is a good source of energy and motivation for everyone.

To sum it up

While knowing there are a lot more key drives and concepts, I think it’s easier to get on track with these 4 when starting the path of Engineering Manager. If one was to stand out, I would say “being an avid learner” as it could continually impact you at a ground level.

Have you ever written a report without much conviction, drowned in hundreds of lines and sentences, each one more formal than the previous one. Tried to make an idea or a concept understood without much success through incomprehensible notes?

Sketchnoting is a restitution method belonging to a group of techniques called "Graphic Facilitation". Very close to the mind mapping method, sketchnoting offers a wider choice of results by mixing sentences, words, and illustrations. Its particularity is that it uses several senses such as hearing, sight, and touch, which makes it very interesting to practice and accessible to a large number of people. The iconographic vocabulary is very personal and you can develop and enrich it as you practice sketching.

“Sketchnoting, also commonly referred to as visual notetaking, is the creative and graphic process through which an individual can record their thoughts with the use of illustrations, symbols, structures, and texts.” Wikipedia

Originally used in "live": during meetings, interviews, conferences, this method can be used after taking notes leaving the sketch artist the possibility to pose his plan.

Why is it so great to use this method in companies?
First of all, anyone can do it. You won't be asked to be a Picasso! The idea of the Sketchnote is to allow everyone to convey ideas effectively through drawings and text.

The basics are a few geometric shapes, which you'll see a little further on in this article... and that's it!

This also allows you to :

• Make reports much more digestible and visual to all the teams.
• Make concepts or ideas more easily understood, by imaging the thought process.
• Deliver a very personal interpretation during the restitution.

And now, to your pencils!

For 1 person, you'll need the following items:

📑 Blank sheets of paper, ✍️Pens, 🖍️ Multi-coloured felt-tip pens

You will also be able to do your sketch on procreate available at the moment only on Ipad.
You will also need to draw graphic elements such as squares, triangles, circles, lines and points. Remember, you are not asked to be an artist, it is the understanding of the idea that counts.

1 - Start by prioritizing your text, sorting out what seems important to you or not. If you are taking notes live, try to divide up the speech you are listening so that you have a plan in mind.

2 - Write your title to structure your summary, using a banner for example.

3 - Link ideas together using linking elements. You can also separate them thanks to separation elements.

4 - Add illustrations to make the drawing easier to understand! Don't panic, you don't have to be Picasso to make a sketch.

5 - Finally, don't forget to play with different colours to highlight important elements.

So you will have understood that this method makes it possible to :

• Text synthesize
• Prioritize information
• Sort information
• Develop your creativity
• Memorize items up to +6.5 times the normal value.

Now you have all the keys in your hands! Now all you have to do is get started on your first sketchnoting...and now, to your pencils! ❤️