Lesson 2: Building a Standard Data Profile

Why you need a profile before you need a service

Most bad storage decisions start with someone saying "we need something fast, safe and cheap" and then picking whichever service they used last time. A data profile replaces those adjectives with numbers. It is a one-page spec sheet for your data. In every service lesson later in the course you will hold that sheet up against the service's documented limits, and the services that do not fit will drop out.

Think of it as the brief you give a removals firm. They need to know cubic metres, stairs or a lift, fragile items and the deadline. They do not need to hear "lots of stuff, be careful."

The ten parts

Walkthrough: a template you can fill in

Keep the profile as YAML in the repo next to the infrastructure code, so it gets reviewed and versioned like code.

# data-profile.yaml for one dataset, not the whole application
name: invoice-pdfs
size:
  current_tib: 4.2            # measured, not guessed
  growth_tib_per_month: 0.3
  object_count: 38_000_000
  typical_object_kib: 120
access:
  type: object                # object | block | file
  protocol: s3-api            # s3-api | nvme | iscsi | nfs | smb | lustre
performance:
  read_iops_peak: 400
  write_iops_peak: 50
  throughput_mb_s_peak: 60
  p99_latency_ms: 200         # a person waiting for a download
pattern:
  read_write_ratio: "8:1"
  access: random
  hot_fraction: 0.05          # 5% of the data is read in any given month
resilience:
  rpo_minutes: 15             # maximum data loss you accept
  rto_minutes: 240            # maximum time until service is back
  survive_az_loss: true
consistency: read-after-write # new invoice must be downloadable at once
sharing:
  concurrent_clients: 200     # web servers plus a batch job
retention:
  keep_days: 3650             # 10 years
  immutable: true             # WORM: nobody may delete early
residency:
  allowed_regions: [eu-west-2]
  regimes: [UK GDPR]
cost:
  budget_usd_month: 400
  dominant: capacity          # small and cold, so storage cost per GiB matters most

Gotcha: profile each dataset, not each application. One app usually has a database (block), user uploads (object) and maybe a shared config folder (file). Each of them gets its own profile.

Access type: the filter that runs first

The protocol your software speaks is a hard constraint, so you check it before anything else. It is not a preference.

Here is the filter as code:

# Simplified: real lessons will add the limits for each service
SERVICES = {
    "S3": {"s3-api"},
    "EBS": {"nvme"},
    "EFS": {"nfs"},
    "FSx Windows": {"smb"},
    "FSx ONTAP": {"nfs", "smb", "iscsi", "nvme-tcp"},
    "FSx Lustre": {"lustre"},
}
need = "smb"
print([s for s, p in SERVICES.items() if need in p])
# -> ['FSx Windows', 'FSx ONTAP']   four of the six are gone before price comes up

That is why price comes later. A legacy Windows app that opens \\\\server\\share cannot use S3, however cheap S3 is, unless you rewrite the app or put a gateway in front of it. A rewrite or a gateway has its own cost, and it belongs in the profile as well.

Turning adjectives into numbers

Once a field is a number you can check it. Say a database needs 5,000 IOPS on a block volume. The current EBS documentation says gp3 includes 3,000 IOPS and 125 MiB/s at any size, and you can provision more, up to 80,000 IOPS and 2,000 MiB/s. So gp3 fits, but you must provision the extra 2,000 IOPS, and they cost money. You can only reach that conclusion once "fast" has become a number.

Best practice: use p99, not the average. An average of 2 ms can hide a 1% tail of 300 ms reads, and that tail is what users notice.

Trade-off: being strict costs money. Ten-year immutability, multi-AZ placement and sub-millisecond latency each narrow the field and raise the bill. Record whether a requirement is a must or a nice to have. The scorecard later in the course uses that distinction.

Key takeaways

• A data profile has ten parts: size and growth, access type and protocol, performance, read/write pattern, durability and availability, consistency, sharing, retention and immutability, residency and compliance, and cost model.
• Every field is a number or a firm yes/no: TiB, TiB per month, IOPS, MB/s, p99 ms, a read:write ratio, RPO and RTO in minutes, retention in days.
• Access type comes first. The S3 API means object storage, NVMe or iSCSI means block, and NFS, SMB or Lustre means file. That filter removes most services before anyone compares prices.
• Translate "fast", "safe" and "cheap" by asking for whom and against what, then check the numbers against documented limits.
• Profile each dataset separately and keep the profile in version control.