Merge branch 'develop' for release v3.35.1

INSTALL.md

@@ -105,7 +105,7 @@ You can define which authentication backend to be used thanks to the `authentica
 * `internal` (default): _LMS_ uses an internal database to store users and their associated passwords (salted and hashed using [Bcrypt](https://en.wikipedia.org/wiki/Bcrypt)). Only the admin user can create, edit or remove other users.
 * `PAM`: the user/password authentication request is forwarded to PAM (see the default [PAM configuration file](conf/pam/lms) provided).
 * `http-headers`: _LMS_ uses a configurable HTTP header field, typically set by a reverse proxy to handle [SSO](https://en.wikipedia.org/wiki/Single_sign-on), to extract the login name. You can customize the field to be used using the `http-headers-login-field` option.
-__Note__: the first created user is the admin user
+__Note__: the first created user is the admin user.
 #### `internal` backend: reset admin password
 Open the the database file located in `/var/lms/lms.db` using `sqlite3`:
 ```sh
@@ -158,6 +158,7 @@ server {
     }
 }
 ```
+__Note__: to mitigate brute force login attempts, _LMS_ uses an internal login throttler based on the client IP address. The `Client-IP` or `X-Forwarded-For` headers are used to determine the real IP adress, so make sure to properly configure your reverse proxy to filter or even erase the values.
 # Run
 ```sh
 systemctl start lms

README.md

@@ -8,24 +8,24 @@ A [demo instance](http://lms-demo.poupon.dev) is available. Note the administrat
 
 ## Main features
 * Recommendation engine
-* Audio transcode for maximum interoperability and low bandwith requirements
+* Audio transcoding for maximum interoperability and reduced bandwith requirements
-* Multi-value tags: `artist`, `genre`, `composer`, `lyricist`, `mood`, `performer`, ...
+* Multi-value tags: `genre`, `albumgenre`, `mood`, `albummood`, `albumgrouping`, ...
+* Artist relationships: `composer`, `conductor`, `lyricist`, `mixer`, `performer`, `producer`, `remixer`
 * [MusicBrainz Identifier](https://musicbrainz.org/doc/MusicBrainz_Identifier) support to handle duplicated artist and release names
 * [ListenBrainz](https://listenbrainz.org) support for:
   * Scrobbling and synchronizing listens
   * Synchronizing 'love' feedbacks
 * ReplayGain support
-* User management, with several authentication backends, see [Deployment](INSTALL.md#deployment)
+* User management, with several [authentication backends](INSTALL.md#authentication-backend)
 * Subsonic API
 
 ## Music discovery
 _LMS_ provides several ways to help you find the music you like:
-* Tag-based filters (ex: _Rock_, _Metal_ and _Aggressive_, _Electronic_ and _Relaxed_, ...)
+* Tag-based filters (ex: "_Rock_", "_Metal_ and _Aggressive_", "_Electronic_ and _Relaxed_", ...)
 * Recommendations for similar artists and albums
-* Radio mode, based on what is in the current playqueue
+* Radio mode, with endless filling of the play queue with tracks similar to what is there
 * Searches in album, artist and track names (including sort names)
 * Starred Albums/Artists/Tracks
-* Various tags to help you filter your music: _mood_, _albummood_, _albumgenre_, _albumgrouping_, ...
 * Random/Starred/Most played/Recently played/Recently added for Artist/Albums/Tracks, allowing you to search for things like:
   * Recently added _Electronic_ artists
   * Random _Metal_ and _Aggressive_ albums
@@ -33,17 +33,8 @@ _LMS_ provides several ways to help you find the music you like:
   * Starred _Jazz_ albums
   * ...
 
-The recommendation engine uses two different sources:
-1. Tags that are present in the audio files
-2. Acoustic similarities of the audio files, using a trained [Self-Organizing Map](https://en.wikipedia.org/wiki/Self-organizing_map)
-
-__Notes on the self-organizing map__:
-* training the map requires significant computation time on large collections (ex: half an hour for 40k tracks using a Core i5)
-* audio acoustic data is pulled from [AcousticBrainz](https://acousticbrainz.org/). Therefore your audio files _must_ contain the [recording](https://musicbrainz.org/doc/Recording) [MusicBrainz Identifier](https://musicbrainz.org/doc/MusicBrainz_Identifier).
-* to enable the audio similarity source, you have to enable it first in the administration panel.
-
 ## Subsonic API
-The API version implemented is 1.16.0 and has been tested on _Android_ using _Subsonic Player_, _Ultrasonic_ and _DSub_.
+The API version implemented is 1.16.0 and has been tested on _Android_ using _Subsonic Player_, _Ultrasonic_, _Symfonium_, and _DSub_.
 Since _LMS_ uses metadata tags to organize music, a compatibility mode is used to browse the collection when using the directory browsing commands.
 The Subsonic API is enabled by default.
 
@@ -53,9 +44,11 @@ __Note__: since _LMS_ may store hashed and salted passwords or may forward authe
 _LMS_ relies exclusively on tags to organize your music collection.
 
 ### Filtering
-You can specify the tags you want to be used to filter your collection. By default, `GENRE`, `ALBUMGROUPING`, `MOOD` and `ALBUMMOOD` tags are used.
+You can specify the tags you want to use to filter your collection. By default, `genre`, `albumgrouping`, `mood` and `albummood` tags are used.
 In the administration  panel, you can set whatever tags you want, even custom tags.
 
+__Note__: you can use the `lms-metadata` tool to have an idea of the tags parsed by _LMS_ using [TagLib](https://github.com/taglib/taglib).
+
 ### Multiple album artists
 _LMS_ requires the `ALBUMARTISTS` and `ALBUMARTISTSSORT` tags to properly handle multiple album artists on the same album. As they are a custom tags, you may need to setup your favorite tagger to add them.
 
@@ -70,12 +63,6 @@ $setmulti(albumartistssort,%_albumartists_sort%)
 * Previous track: <kbd>Ctrl</kbd> + <kbd>Left</kbd>
 * Next track: <kbd>Ctrl</kbd> + <kbd>Right</kbd>
 
-## Security considerations
-_Wt_ (the web framework used) has some [built-in security measures](https://www.webtoolkit.eu/wt/features#security), but _LMS_ also has some too:
-* to mitigate brute force login attempts, _LMS_ uses an internal login throttler based on the client IP address. The `Client-IP` or `X-Forwarded-For` headers are used to determine the real IP adress, so make sure to properly configure your reverse proxy to filter or even erase the values (see example in [INSTALL.md](INSTALL.md)).
-* all passwords are stored hashed and salted using [bcrypt](https://fr.wikipedia.org/wiki/Bcrypt)
-* all the resources relative to the music collection (tracks, covers, etc.) are private to an anthenticated session
-
 ## Installation
 See [INSTALL.md](INSTALL.md) file.
 

approot/admin-database.xml

@@ -34,6 +34,7 @@
 			</div>
 		</div>
 		<legend>${tr:Lms.Admin.Database.scan-options}</legend>
+		${<if-has-recommendation-engine>}
 		<div class="col-lg-6">
 			<label class="form-label" for="${id:recommendation-engine-type}">
 				${tr:Lms.Admin.Database.recommendation-engine-type}
@@ -43,6 +44,7 @@
 				${recommendation-engine-type-info}
 			</div>
 		</div>
+		${</if-has-recommendation-engine>}
 		<div class="col-12">
 			<label class="form-label" for="${id:clusters}">
 				${tr:Lms.clusters}

src/libs/services/auth/impl/LoginThrottler.cpp

@@ -54,7 +54,7 @@ LoginThrottler::removeOutdatedEntries()
 
 	for (auto it {std::begin(_attemptsInfo)}; it != std::end(_attemptsInfo); )
 	{
-		if (it->second <= now)
+		if (it->second.nextAttempt <= now)
 			it = _attemptsInfo.erase(it);
 		else
 			++it;
@@ -65,7 +65,6 @@ void
 LoginThrottler::onBadClientAttempt(const boost::asio::ip::address& address)
 {
 	const boost::asio::ip::address clientAddress {getAddressToThrottle(address)};
-
 	const Wt::WDateTime now {Wt::WDateTime::currentDateTime()};
 
 	if (_attemptsInfo.size() >= _maxEntries)
@@ -73,9 +72,25 @@ LoginThrottler::onBadClientAttempt(const boost::asio::ip::address& address)
 	if (_attemptsInfo.size() >= _maxEntries)
 		_attemptsInfo.erase(Random::pickRandom(_attemptsInfo));
 
-	_attemptsInfo[address] = now.addSecs(3);
+	AttemptInfo& attemptInfo {_attemptsInfo[address]};
+	if (attemptInfo.nextAttempt.isValid())
+	{
+		assert(attemptInfo.nextAttempt <= now); // should not be called if throttled
+		attemptInfo = {};
+	}
 
-	LMS_LOG(AUTH, DEBUG) << "Registering bad attempt for '" << clientAddress.to_string() << "'";
+	attemptInfo.badConsecutiveAttemptCount += 1;
+
+	LMS_LOG(AUTH, DEBUG) << "Registering bad attempt for '" << clientAddress.to_string() << "', consecutive bad attempts count = " << attemptInfo.badConsecutiveAttemptCount;
+	if (attemptInfo.badConsecutiveAttemptCount >= _maxBadConsecutiveAttemptCount)
+	{
+		LMS_LOG(AUTH, DEBUG) << "Throttling '" << clientAddress.to_string() << "'";
+		attemptInfo.nextAttempt = now.addMSecs(std::chrono::duration_cast<std::chrono::milliseconds>(_throttlingDuration).count());
+	}
+	else
+	{
+		attemptInfo.nextAttempt = {};
+	}
 }
 
 void
@@ -95,7 +110,10 @@ LoginThrottler::isClientThrottled(const boost::asio::ip::address& address) const
 	if (it == _attemptsInfo.end())
 		return false;
 
-	return it->second > Wt::WDateTime::currentDateTime();
+	if (!it->second.nextAttempt.isValid())
+		return false;
+
+	return it->second.nextAttempt > Wt::WDateTime::currentDateTime();
 }
 
 } // Auth

src/libs/services/auth/impl/LoginThrottler.hpp

@@ -19,8 +19,7 @@
 
 #pragma once
 
-#include <shared_mutex>
+#include <chrono>
-#include <string>
 #include <unordered_map>
 
 #include <Wt/WDateTime.h>
@@ -44,8 +43,15 @@ namespace Auth
 			void removeOutdatedEntries();
 
 			const std::size_t _maxEntries;
+			static constexpr std::size_t _maxBadConsecutiveAttemptCount {5};
+			static constexpr std::chrono::seconds _throttlingDuration {3};
 
-			std::unordered_map<boost::asio::ip::address, Wt::WDateTime>	_attemptsInfo;
+			struct AttemptInfo
+			{
+				Wt::WDateTime nextAttempt;
+				std::size_t badConsecutiveAttemptCount{};
+			};
+			std::unordered_map<boost::asio::ip::address, AttemptInfo>	_attemptsInfo;
 	};
 } // Auth
 

src/libs/services/scanner/CMakeLists.txt

@@ -1,6 +1,5 @@
 
 add_library(lmsscanner SHARED
-	impl/AcousticBrainzUtils.cpp
 	impl/ScannerService.cpp
 	impl/ScannerStats.cpp
 	)

src/libs/services/scanner/impl/AcousticBrainzUtils.cpp

@@ -1,87 +0,0 @@
-/*
- * Copyright (C) 2018 Emeric Poupon
- *
- * This file is part of LMS.
- *
- * LMS is free software: you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation, either version 3 of the License, or
- * (at your option) any later version.
- *
- * LMS is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with LMS.  If not, see <http://www.gnu.org/licenses/>.
- */
-
-#include "AcousticBrainzUtils.hpp"
-
-#include <boost/property_tree/ptree.hpp>
-#include <boost/property_tree/json_parser.hpp>
-
-#include <Wt/WIOService.h>
-#include <Wt/Http/Client.h>
-
-#include "utils/IConfig.hpp"
-#include "utils/Logger.hpp"
-#include "utils/Service.hpp"
-#include "utils/UUID.hpp"
-
-
-namespace AcousticBrainz
-{
-
-static
-std::string
-getJsonData(const UUID& mbid)
-{
-	static constexpr std::string_view defaultAPIURL {"https://acousticbrainz.org"};
-
-	const std::string url {std::string {Service<IConfig>::get()->getString("acousticbrainz-api-base-url", defaultAPIURL)} + "/api/v1/" + std::string {mbid.getAsString()} + "/low-level"};
-
-	boost::asio::io_service ioService;
-
-	Wt::Http::Client client {ioService};
-	client.setFollowRedirect(true);
-	client.setSslCertificateVerificationEnabled(true);
-	client.setMaximumResponseSize(256*1024);
-
-	if (!client.get(url))
-	{
-		LMS_LOG(DBUPDATER, ERROR) << "Cannot perform a GET request to url '" << url << "'";
-		return {};
-	}
-
-	std::string response;
-	client.done().connect([&](Wt::AsioWrapper::error_code ec, const Wt::Http::Message &msg)
-	{
-		if (ec)
-		{
-			LMS_LOG(DBUPDATER, ERROR) << "GET request to url '" << url << "' failed: " << ec.message();
-			return;
-		}
-
-		if (msg.status() != 200)
-		{
-			LMS_LOG(DBUPDATER, ERROR) << "GET request to url '" << url << "' failed: status = " << msg.status() << ", body = " << msg.body();
-			return;
-		}
-
-		response = msg.body();
-	});
-
-	ioService.run();
-
-	return response;
-}
-
-std::string
-extractLowLevelFeatures(const UUID& recordingMBID)
-{
-	return getJsonData(recordingMBID);
-}
-
-} // namespace Scanner::AcousticBrainz

src/libs/services/scanner/impl/AcousticBrainzUtils.hpp

@@ -1,30 +0,0 @@
-/*
- * Copyright (C) 2018 Emeric Poupon
- *
- * This file is part of LMS.
- *
- * LMS is free software: you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation, either version 3 of the License, or
- * (at your option) any later version.
- *
- * LMS is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with LMS.  If not, see <http://www.gnu.org/licenses/>.
- */
-
-#pragma once
-
-#include <string>
-
-class UUID;
-
-namespace AcousticBrainz
-{
-	std::string extractLowLevelFeatures(const UUID& recordingMBID);
-}
-

src/libs/services/scanner/impl/ScannerService.cpp

@@ -38,7 +38,6 @@
 #include "utils/Logger.hpp"
 #include "utils/Path.hpp"
 #include "utils/UUID.hpp"
-#include "AcousticBrainzUtils.hpp"
 
 using namespace Database;
 
@@ -526,7 +525,6 @@ ScannerService::scan(bool forceScan)
 	if (!_abortScan)
 	{
 		checkDuplicatedAudioFiles(stats);
-		fetchTrackFeatures(stats);
 		reloadSimilarityEngine(stats);
 	}
 
@@ -560,85 +558,6 @@ ScannerService::scan(bool forceScan)
 	}
 }
 
-bool
-ScannerService::fetchTrackFeatures(TrackId trackId, const UUID& recordingMBID)
-{
-	std::map<std::string, double> features;
-
-	LMS_LOG(DBUPDATER, INFO) << "Fetching low level features for recording '" << recordingMBID.getAsString() << "'";
-	const std::string data {AcousticBrainz::extractLowLevelFeatures(recordingMBID)};
-	if (data.empty())
-	{
-		LMS_LOG(DBUPDATER, ERROR) << "Track " << trackId.getValue() << ", recording MBID = '" << recordingMBID.getAsString() << "': cannot extract features using AcousticBrainz";
-		return false;
-	}
-
-	{
-		auto uniqueTransaction {_dbSession.createUniqueTransaction()};
-
-		Track::pointer track {Track::find(_dbSession, trackId)};
-		if (!track)
-			return false;
-
-		_dbSession.create<TrackFeatures>(track, data);
-	}
-
-	return true;
-}
-
-void
-ScannerService::fetchTrackFeatures(ScanStats& stats)
-{
-	if (_recommendationServiceType != ScanSettings::RecommendationEngineType::Features)
-		return;
-
-	ScanStepStats stepStats{stats.startTime, ScanProgressStep::FetchingTrackFeatures};
-
-	LMS_LOG(DBUPDATER, INFO) << "Fetching missing track features...";
-
-	struct TrackInfo
-	{
-		TrackId id;
-		UUID recordingMBID;
-	};
-
-	const auto tracksToFetch {[&]()
-	{
-		std::vector<TrackInfo> res;
-
-		auto transaction {_dbSession.createSharedTransaction()};
-
-		auto trackIds {Track::findWithRecordingMBIDAndMissingFeatures(_dbSession, Range {})};
-		for (const TrackId trackId : trackIds.results)
-		{
-			const Track::pointer track {Track::find(_dbSession, trackId)};
-			res.emplace_back(TrackInfo {track->getId(), *track->getRecordingMBID()});
-		}
-
-		return res;
-	}()};
-
-	stepStats.totalElems = tracksToFetch.size();
-	notifyInProgress(stepStats);
-
-	LMS_LOG(DBUPDATER, INFO) << "Found " << tracksToFetch.size() << " track(s) to fetch!";
-
-	for (const TrackInfo& trackToFetch : tracksToFetch)
-	{
-		if (_abortScan)
-			return;
-
-		if (fetchTrackFeatures(trackToFetch.id, trackToFetch.recordingMBID))
-			stats.featuresFetched++;
-
-		stepStats.processedElems++;
-		notifyInProgressIfNeeded(stepStats);
-	}
-
-	notifyInProgress(stepStats);
-	LMS_LOG(DBUPDATER, INFO) << "Track features fetched!";
-}
-
 void
 ScannerService::refreshScanSettings()
 {

src/libs/services/scanner/impl/ScannerService.hpp

@@ -77,8 +77,6 @@ namespace Scanner
 			void scan(bool force);
 
 			void scanMediaDirectory( const std::filesystem::path& mediaDirectory, bool forceScan, ScanStats& stats);
-			bool fetchTrackFeatures(Database::TrackId trackId, const UUID& MBID);
-			void fetchTrackFeatures(ScanStats& stats);
 
 			// Helpers
 			void refreshScanSettings();

src/libs/subsonic/impl/SubsonicResource.cpp

@@ -1883,8 +1883,8 @@ handleGetCoverArt(RequestContext& context, const Wt::Http::Request& /*request*/,
 	if (!trackId && !releaseId)
 		throw BadParameterGenericError {"id"};
 
-	std::size_t size {getParameterAs<std::size_t>(context.parameters, "size").value_or(256)};
+	std::size_t size {getParameterAs<std::size_t>(context.parameters, "size").value_or(1024)};
-	size = Utils::clamp(size, std::size_t {32}, std::size_t {1024});
+	size = Utils::clamp(size, std::size_t {32}, std::size_t {2048});
 
 	std::shared_ptr<Image::IEncodedImage> cover;
 	if (trackId)
@@ -2016,7 +2016,7 @@ static std::unordered_map<std::string, MediaRetrievalHandlerFunc> mediaRetrieval
 	// Media retrieval
 	{"/download",		Stream::handleDownload},
 	{"/stream",			Stream::handleStream},
-	{"/getCoverArt",		handleGetCoverArt},
+	{"/getCoverArt",	handleGetCoverArt},
 };
 
 

src/lms/ui/LmsApplication.cpp

@@ -401,7 +401,7 @@ LmsApplication::createHome()
 R"(function(current) {
 	const menuItems = document.querySelectorAll('.nav-item a[href]:not([href=""])');
     for (const menuItem of menuItems) {
-        if (menuItem.getAttribute("href").indexOf(current) !== -1) {
+        if (menuItem.getAttribute("href") === current) {
             menuItem.classList.add('active');
         }
 		else {

src/lms/ui/ModalManager.cpp

@@ -44,8 +44,6 @@ namespace UserInterface
 	void
 	ModalManager::show(std::unique_ptr<Wt::WWidget> modalWidget)
 	{
-		assert(count() == 0);
-
 		LMS_LOG(UI, DEBUG) << "Want to show, id = " << modalWidget->id();
 
 		std::ostringstream oss;

src/lms/ui/admin/DatabaseSettingsView.cpp

@@ -142,6 +142,13 @@ class DatabaseSettingsModel : public Wt::WFormModel
 			scanSettings.modify()->setClusterTypes(LmsApp->getDbSession(), std::set<std::string>(clusterTypes.begin(), clusterTypes.end()));
 		}
 
+		static ScanSettings::RecommendationEngineType getCurrentRecommendationEngine()
+		{
+			auto transaction {LmsApp->getDbSession().createSharedTransaction()};
+			const ScanSettings::pointer scanSettings {ScanSettings::get(LmsApp->getDbSession())};
+			return scanSettings->getRecommendationEngineType();
+		}
+
 	private:
 		void initializeModels()
 		{
@@ -211,6 +218,9 @@ DatabaseSettingsView::refreshView()
 	t->setFormWidget(DatabaseSettingsModel::UpdateStartTimeField, std::move(updateStartTime));
 
 	// recommendation engine type
+	// Hide the settings if the engine is set to clusters, as we don't want users to switch to acoustic features (currently broken)
+	// Otherwise, give the user a way to switch back to clusters
+	t->setCondition("if-has-recommendation-engine", model->getCurrentRecommendationEngine() == ScanSettings::RecommendationEngineType::Features);
 	auto recommendationEngineType {std::make_unique<Wt::WComboBox>()};
 	recommendationEngineType->setModel(model->recommendationEngineTypeModel());
 	t->setFormWidget(DatabaseSettingsModel::RecommendationEngineTypeField, std::move(recommendationEngineType));