CHANGELOG
THE LATEST UPDATES

v2212.0

• Improve Ceph Upgrade procedure for OpenSUSE systems.

• Fixed a bug when updating CRUSH map while creating OSDs.

• Fixed warning about even number of mons showing up while still creating mons.

• Fixed a rare bug causing unnecessary error messages when creating multiple mons on certain systems.

v2211.0

• Added option to configure the CephFS filesystem name.
  Clusters imported into croit may already have CephFS set up with a different filesystem name, this name can now be configured. The setting is available under the name croit cephfs name on the Settings page.

• Added option to configure the base URI of the management node for download links.
  The setting is available under the name croit base uri on the Settings page.

• Fixed an issue when wiping disks taken from other ceph clusters.

• Fixed an issue with the disk LED button not working.

• Updated the template for Samba with AD to include the NetBIOS names of the servers running Samba gateways.

• Fixed an issue where the wrong systemd service is started on nodes after changing their hostname.

• Fixed an error message in the croit logs when a server doesn't have any disks.

• Increased logging output limit for tasks.
  Some tasks (e.g. creating multiple OSDs) reached the logging output limit in croit before.

• Fixed a rare and incorrect warning about data loss when creating many OSDs.

• Fixed an issues where HA groups with no servers associated with them wouldn't be listed.

• Fixed an issue where RGW pools were created when navigating to the S3 User page, despite S3 not being configured on the cluster.

• Added dialog to ask again for confirmation when simultaneously rebooting servers.

• Added experimental support to upload iSCSI gateway certificates when configuring the iSCSI API in secure mode.
  To enable secure mode, set croit iscsi api secure to true on the Settings page. The certificate files (named iscsi-gateway.crt and iscsi-gateway.key) need to be put into /config/, and will be automatically uploaded to the iSCSI nodes. This feature is still experimental, and may be improved in the future.

• Improved performance when configuring RBD snapshot schedules with many RBDs.

v2209.0

• Croitd daemon now uses HTTPS when connecting to the management node.
  Please note that the croit daemon currently doesn't verify the certificate. We recommend also adding your own certificate, otherwise traffic can easily be decrypted.

• Fixed an issue with the network latency statistics not being collected.

• Added option to select iSCSI target IQN in case multiple targets are configured.

v2208.1

• Fixed an issue when deleting OSDs in the Crush Map view.

• Fixed iSCSI gateways showing as inaccessible/unauthorized in the terminal view on the iSCSI service page.

v2208.0

• Added template for the iscsi gateway configuration file.
  We don't recommend editing this template unless instructed by the croit support team.

• Changed daemon to write CephFS key in a file in /etc/ceph/ for easier remounting of /persistent in hook scripts.

• Added a check if /config/config.yml contains any syntax errors.
  If it does, we fall back to the last working configuration, or the croit defaults.

• Added configuration option for the repository URL to use for ceph upgrades (for openSUSE).
  This is mostly meant to be used for debugging, or when an upgrade to a specific ceph version is required.

• Added configuration option to change target IQN for iSCSI.
  This is only meant to be used by the croit support team to manage specific iSCSI setups.

v2207.1

• Added hook scripts events OnCroitStartup and OnCroitAsyncStartup to run a hook script on the management node during croit startup.

• Added option to keep waiting without asking again to rolling reboot of multiple servers in case the timeout is reached.
  This can also be made default by adding rollingRebootNoTimeout: true to the /config/config.yml file, in which case the rolling reboot will never time out.

• Improved handling of task advisor warnings after completing a ceph upgrade.

v2207.0

• Fixed a configuration error with ctdb (used for samba).

• Rolling reboot will now ask the user if they want to keep waiting or stop the rolling reboot once the timeout is reached.

• Removed unnecessary validation of S3 bucket policies.

• Fixed handling of S3 subuser without any access permissions.

• Multiple performance improvements.
  In particular, we fixed an issue introduced in croit v2201.1.

v2206.2

• Fixed an issue parsing the SMART health status.

• Added endpoint to list S3 subusers directly under GET s3/users/{uid}/subusers.

v2206.1

• Fixed an issue with disks (in particular NVMes with namespaces) not being updated due to longer than expected WWNs.

v2206.0

• Fixed an issue where DB/WAL partitions weren't correctly associated with OSDs in our UI.

• Fixed a timeout issue when turning a large disk into a mon disk.
  mkfs.xfs may take longer than expected on large disks, so the timeout for that was increased.

• Small improvements to auto-updater.

v2205.0

• The croit Pacific (and later) container is now used as the default croit/croit:latest container. Users running an older (pre-pacific) ceph version should update to the croit/croit:latest.octopus container instead. The croit app will give you the option to switch to the correct container in case the Pacific container was started on a pre-pacific ceph cluster.

• Added option to upgrade to Ceph Quincy.
  Warning: We still don't recommend migrating to Quincy (nor Pacific), due to stability issues!

• Added Discord notification script to hook script library.

• Added support for specific DB/WAL setups using LVM instead of partitions.

• Added a small improvement to the Ceph Pacific upgrade procedure.

• Fixed a memory corruption issue in the croitd daemon.

• Fixed the utilization on the status page to display the maximum utilization per OSD for each device class, instead of an average over OSDs.

• Fixed an issue where joining a samba domain failed when the password contains special bash characters.

• Fixed an issue where NVMes with multiple namespaces weren't detected as separate disks.

• Fixed a compatibility issue with old debian stretch images.

• Fixed an issue when doing a minor upgrade to Ceph 16.2.7.

• Fixed an issue of the OSD state statistics not being collected in Ceph Octopus or later.

• Fixed s3:* not being allowed when configuring bucket policies.

• Removed ceph configuration defaults setting active hours for osd scrubs (osd scrub begin/end hour).

v2202.0

• Fixed not being able to add more iSCSI gateways on clusters where the iSCSI services take too long to start.
  Adding more gateways may have failed on such clusters if the other gateways weren't healthy fast enough.

• Fixed an issue that caused a pool .rgw.root to be created despite not using RGW.

• Fixed a missing animation when creating OSDs.

• Improved performance of the pools page.

• Added some small improvements to the Ceph Pacific upgrade procedure.

v2201.1

• Added an authenticated endpoint to access file on the management node.
  Similar to the existing endpoint serving files in /config/extra-files (inside the croit docker container), the new endpoint serves files from the /config/extra-files-secure folder. The API endpoint is reachable under /extra-files-secure/path-to-file similar to the /extra-files/... endpoint, but needs to be authenticated against using a croit user account with the Secure folder account role (via HTTP Basic authentication).

• Fixed an issue that caused some MDS daemons to crash.
  The last croit update enabled the messenger v2 protocol for MDS. This can result in the MDS daemons crashing. Messenger v2 is disabled for now if all MDS daemons are not already using v2.

• Fixed S3 user stats missing for users that contain '@' in their user ID.

• Fixed S3 stats missing for custom zones and placement rules.
  The croit-rgw-monitoring ceph key will now be updated to include any necessary RGW pools according to zones and placement rules. If you want to prevent croit from updating this key, and instead manage it manually, add disableRgwMonitoringKeyUpdate: true to your /config/config.yaml inside the croit docker container.

• Fixed not being able to edit a user's roles without also changing his password.

• Added an option to configure the default command timeout.
  To do so, add defaultCommandTimeout: 8 (here for example 8 seconds, the current default) to your /config/config.yaml inside the croit docker container. Generally speaking, this shouldn't be necessary and if you see many timeout warnings in the frontend, please contact croit support (support@croit.io) first.

• Added some improvements to the Pacific upgrade to make it more stable.

v2201.0

• Added option to upgrade to Ceph Pacific.
  Warning: We still don't recommend migrating to Pacific, due to stability issues! Due to compatibility issues between Pacific and older ceph versions (in particular with RGW), we now offer a separate croit container for Pacific clusters. The new container is currently tagged as latest.pacific, and will in the future replace the latest container.
  The Pacific upgrade procedure will try to switch to the new container for you. Should that fail, you can also simply update the croit container to latest.pacific manually.
  If you intend to stay on an older ceph release, please use the latest.octopus tag from now on, until you decide to upgrade to Pacific. Don't worry if you start the Pacific container by accident, certain features will be blocked, and you will be given the option to switch to the Octopus container again.
  Starting with Pacific, our default OS is now openSUSE instead of Debian. After the upgrade, the latest openSUSE image will be selected as new default for your servers. Please update your hook scripts and fix OS-specific commands (e.g. apt-get).
  If you have any issues or questions, or need help with any of the upgrade steps, please contact croit support (support@croit.io) for further assistance.

• Fixed an issue that caused scheduled jobs (such as automatic updates and backups or scheduled snapshots) to not run anymore.
  For users affected by this issue, it also caused log files not being cleaned properly in the croit container, filling up the disk on the management node.
  As a temporary workaround for this issue in older croit versions, you can simply restart the croit docker container.

• Added toggle for base2 filesizes to RBD view.

• Added option for viewer users to change their own password.

• Added an extra warning when changing RBD image name or features and improved some error messages when deleting RBD images.

• Fixed RGW user stats not being collected and missing from the UI.

• Fixed an issue where servers incorrectly report being out-of-sync with the management node clock and having large clock drifts.

• Fixed setting the server-specific variables (e.g. hostname) of the Samba AD domain join templates.
  The variables now have the correct values for the server joining the domain.

• Fixed default ceph.conf options to have messenger v2 (msgr2) enabled.

• Fixed rolling reboot for servers when switching between Debian and openSUSE.

• Fixed an issue where servers would show as offline in the croit UI when they had broken NFS connections.

• Fixed an issue where a warning about failed backups incorrectly showed up.

v2111.3

• Added a template for the Samba AD domain join.
  This template can be edited to customize the commands used to join the domain.

• Added a warning in case the backup on mon disks fails.

• Fixed an issue in the croitd daemon when reporting server state on nodes that have no disks.

• Fixed an issue when adding or changing NFS exports.
  Our daemon will now tell NFS Ganesha to reload the new configuration.

• Fixed an issue when monitoring RBD images not in the default rbd pool.

• Fixed not being able to create an RBD image with no features enabled, and removing all features on existing RBD images.

• Fixed exclusive lock being enabled when adding RBD images as disks to iSCSI on suse clusters.

• Fixed sorting by Usage Objects in the S3 explorer.

• Fixed an issue with mismatching timezones for certain tasks.

• Changing SSH access keys will now not also try to update the IPMI password on servers if the password wasn't changed.

• Small improvements to the upgrade procedure for ceph.

• Fixed an issue with the network setup script and multiple NICs with the same name on the same node.

• Fixed default gateways of VLANs not being set up correctly.

v2111.2

• Fixed an issue in the croitd daemon when detecting the model name of NVMe disks.

v2111.1

• Fixed an issue when detecting different disks with the same WWN in the croitd daemon.
  Some disks may have the same WWN with different vendor extensions. The daemon handled this incorrectly before.

v2111.0

• Switched to a new network configuration script to set up the network on the cluster nodes.
  This setup script is available as the template network-setup and can be edited for custom networks configurations. Should you experience any network issues after upgrading to the new version, please contact croit support. You can disable the new script by adding useNetworkSetupScript: false to the configuration file /config/config.yml inside the docker container.

• Improved performance of disk detection by parallelizing it.

• Fixed upgrade procedure for MDS services when migrating to a new ceph version.

• Fixed not being able to select an EC pool as data pool when configuring a S3 placement rule.

• Fixed an issue in the configuration dialog for bond interfaces.
  The LACP checkbox wasn't pre-selected correctly according to current settings.

• Fixed an incorrect warning about long hardware detection times.

• Improved stability in the new daemon while the management node is down.

v2110.0

• Replaced the daemon running on croit nodes with a newer version.
  Our croit daemon running on the nodes was rewritten for the future croit platform. If you notice any problems, please let us know.

• Added experimental support for IPoIB interfaces.
  MAC addresses of such interfaces are 20 bytes long. We use the last 8 bytes for identification, as the first 12 bytes can change over time.

• Added the option to edit S3 keys of a user.
  Small changes were made to the S3 subuser API (they are put under a /subusers resource to prevent naming conflicts).

• Fixed a timeout issue when deleting or flattening large RBD images.

• Added a new option disableDhcp in config.yml.
  When set to true, the DHCP server in the docker container will be disabled.

• Fixed an issue where SMB/NFS services that aren't associated with any server weren't displayed and couldn't be deleted or modified.

• Fixed an issue with pagination in the S3 explorer.

• Fixed rgw pools not using the correct crush rule when created.

• Fixed displaying the ssh key in the server access dialog.

• Fixed debian update failing because of testing name change.

• Added a check when creating a bond for the length of its name (max. 15 characters).

• Fixed an issue when doing a rolling restart of SMB services.

• Removed LXC container support.

v2108.0

• Fixed an issue when wiping encrypted OSDs that also use a DB/WAL device.
  We urge anyone with OSDs with separate DB/WAL devices that used to be encrypted (or still are) to update to the latest croit version without rebooting any servers. The new update will display a warning if your OSDs are affected by this issue. If so, please contact croit support and do not reboot any server.

• Added a new page for managing S3 placement rules.
  The new page can be accessed under the S3 Explorer tab. Here you can configure new placement rules and add storage classes to these rules. The default placement for individual users can also be configured on the users page. Please note that creating buckets in the croit UI will either use the specified placement rule, or the default rule of the owner.
  The API for s3 placements has also changed. If you used this API before, please update your requests or contact croit support for help.

• Added task advisor warning for pre-Octopus EOL.

• Added a template for DNS configuration (/etc/resolv.conf).
  This fixes an issue where the DNS configuration wasn't properly applied to the croit nodes.

• Added a template for the keepalived configuration (used for HA groups).

v2106.1

• Removed milliseconds from scheduled CephFs and RBD snapshots.
  Scheduled snapshots were postfixed with the current time in the format "yyyy-MM-dd.HH:mm:ss.SSS". We removed the milliseconds (SSS) from this format for better compatibility with shadow copies in Samba.

• Added descriptions to template variables.

• Added a task advisor warning when multiple default gateways are configured.

v2106.0

• Users with the Viewer role are now allowed to view the CephFS explorer.
  Viewers are only allowed to browse and download files, they can't upload or change any settings.

• Updated the nfs-ganesha template for better performance.
  We suggest using the updated default template for the ganesha config.

• Added option to configure index and extra pool for RGW placement rules.

• Removed ctdb-samba-stretch template.

• Fixed an issue where some pools weren't protected when created.

• Fixed an issue where trying to change compression settings of a pool failed with a 404 error.

• Added a warning when switching between openSUSE and debian images.

• Added a task advisor warning when disks show SMART health issues.

v2105.0

• Fixed an issue where a rebooting server wasn't recognized and added as a new server instead.

• Fixed an issue when managing the ceph dashboard.

• Fixed an issue when cloning RBD snapshot on Mimic.

• Fixed an issue when enabling RBD mirroring.

v2104.1

• Fixed an issue where nodelete flag was set too often on pools.

v2104.0

• The kernel commandline template has been updated.
  Any previous changes made to the template are stored as a new preset "custom-old". The new template is required for openSUSE images.

• Added support for scrubbing pools.
  When selecting a pool, you can now use the "Scrub" button to start a scrub operation on the pool.

• Added configurable timezones for snapshot schedules.
  The timezone for schedules can be configured in the snapshot schedule tab.

• Added new template for the apt proxy configuration.

• Added support for S3 bucket versioning.
  Please note that once versioning is enabled for a bucket, it can't be disabled anymore, only suspended.

• Fixed handling of symlink files in CephFS explorer.
  Support for symlink files is still being improved.

v2103.0

• Experimental support for openSUSE.
  Added support for our croit images based on openSUSE instead of debian. These images are currently only available as nightly images. Debian and openSUSE images should not be mixed on the same cluster.

• Removed hook script parameter for Pre/PostNetworkConfig hooks.
  Due to changes with our network setup procedure, the parameters for changed NICs has been removed.

• Added option for multiple active MDS services.
  When selecting an MDS service, click on Active MDS to choose the number of MDS services that ceph should keep active.

• Added support to configure per-pool BlueStore compression.
  BlueStore compression can be configured by selecting a pool and clicking on Compression.

• Added warning when deleting RGW pools with existing placement rules.

• Added warning when choosing a default network for automatic ping metrics that doesn't contain all servers.

v2102.2

• Support separate DB and WAL devices.
  When creating OSDs, DB and WAL devices can now be specified separately. Previously, croit only supported specifying a single device for both; it would use partition size to determine whether to store only the WAL or both DB and WAL on the device.

• Remove kernel commandline dracut template.
  croit doesn't provide CentOS images anymore; this removes support for booting CentOS entirely.

• Fall back to iproute2 if no MAC is found.
  This change allows InfiniBand NICs in IPoIB mode to work with croit.

• Make bash history upgrade proof and add timestamps to bash history.

• Add /api/audit API.

v2102.0

• Wipe disks on all servers in parallel.
  When wiping disks in multiple servers, each server can now wipe a disk at the same time.

• Fix LDAP query for account name.

• Add Task Advisor warning for non-highly-available MDS setups.
  croit will now warn when there are too few standby/standby-replay MDS.

• Prevent removing admin role from only admin account.
  croit will prevent users from locking themselves out of their croit instance. To restore the initial admin account (username and password admin) use the /root/restore_admin.sh script.

• Add destroy flag to /api/disks/wipe.
  This allows reusing an OSD ID when wiping an OSD.

v2101.1

• Make LDAP search for groups configurable.
  There are new groupClassName and memberAttributeName options in config.yml.

• Fix LDAP search request for OpenLDAP.

• Add initial hosts to CRUSH map.
  croit will now add hosts to the CRUSH map during cluster creation.

v2101.0

• Refactor MDS output in cluster status.

• Set default ceph release to Octopus.

• Detect current ceph release when importing cluster.

• Fix rolling service restart for OSDs.

• Monitor OSD fragmentation in Task Advisor and disk view.

• Add API support for ceph osd pool deep-scrub <pool>.

v2012.1

• Add option for disabling CPU idle states.

• Add "OSD Statistics" view.
  This view replaces the read/write/subop latency and data balance graphs for improved performance on large clusters.

• Add experimental LXC container support.

• Set pg_num_min when creating a pool.

• Fix memory hungry CRUSH map view on large clusters.

• Fix invalid PW/username for iSCSI client.

• Make Samba virtual IPs work without creating a network in croit.

• Fix old ceph configs still containing underscores in their name.

• Only activate OSDs 120s after boot.

• Add OSD pause flag in Maintenance view.

• Show management node in ping metrics.

v2012.0

• Fix storing bond MAC addresses.
  Previously bond MACs were stored improperly allowing only one of the bonded NICs to be used for booting.

• Support Samba with sssd.

• Restart nfs service when changing filesystem path of an export.

• Check for OSDs running ShallowFSCK when upgrading ceph.

• Remove support for S3 over NFS.

• Allow editing MAC addresses.
  This feature can be used when replacing NICs in a server while the server is turned off.

• Store backup on MONs.
  croit will now store a daily, unencrypted backup on all MON hosts in /var/lib/ceph/mon/.croit-backup.zip.

• Restart dhcpd when DHCP template is updated.

• Add view for rbd perf image iostat.

• Implement S3 bucket policy management.

• Clean graphite stats after two weeks.

• Recreate default CRUSH rule after creating OSDs.

• Set MDS to Active/Hot/Cold configuration by default.

• Fix dmidecode RAM parsing.

• Sort Task Advisor tasks by severity.

• Add OnRbdSnapshotComplete hook.

• Experimental support for podman.

• Fix iSCSI exports in non-default network.

• Show targets column in Snapshot Schedule view.

• Make templates available during cluster setup.