owen/lms
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'develop' for release v3.35.1

Browse Source
This commit is contained in:
emeric
2023-01-21 14:38:14 +01:00
parent 6898a4096a 93fac1a0ea
commit 1b9cd9e14b
14 changed files with 60 additions and 239 deletions
Download Patch File Download Diff File Expand all files Collapse all files
INSTALL.md
+2 -1
View File
@@ -105,7 +105,7 @@ You can define which authentication backend to be used thanks to the `authentica
* `internal` (default): _LMS_ uses an internal database to store users and their associated passwords (salted and hashed using [Bcrypt](https://en.wikipedia.org/wiki/Bcrypt)). Only the admin user can create, edit or remove other users.
* `PAM`: the user/password authentication request is forwarded to PAM (see the default [PAM configuration file](conf/pam/lms) provided).
* `http-headers`: _LMS_ uses a configurable HTTP header field, typically set by a reverse proxy to handle [SSO](https://en.wikipedia.org/wiki/Single_sign-on), to extract the login name. You can customize the field to be used using the `http-headers-login-field` option.
__Note__: the first created user is the admin user
__Note__: the first created user is the admin user.
#### `internal` backend: reset admin password
Open the the database file located in `/var/lms/lms.db` using `sqlite3`:
```sh
@@ -158,6 +158,7 @@ server {
}
}
```
__Note__: to mitigate brute force login attempts, _LMS_ uses an internal login throttler based on the client IP address. The `Client-IP` or `X-Forwarded-For` headers are used to determine the real IP adress, so make sure to properly configure your reverse proxy to filter or even erase the values.
# Run
```sh
systemctl start lms
README.md
+10 -23
View File
@@ -8,24 +8,24 @@ A [demo instance](http://lms-demo.poupon.dev) is available. Note the administrat
## Main features
* Recommendation engine
* Audio transcode for maximum interoperability and low bandwith requirements
* Multi-value tags: `artist`, `genre`, `composer`, `lyricist`, `mood`, `performer`, ...
* Audio transcoding for maximum interoperability and reduced bandwith requirements
* Multi-value tags: `genre`, `albumgenre`, `mood`, `albummood`, `albumgrouping`, ...
* Artist relationships: `composer`, `conductor`, `lyricist`, `mixer`, `performer`, `producer`, `remixer`
* [MusicBrainz Identifier](https://musicbrainz.org/doc/MusicBrainz_Identifier) support to handle duplicated artist and release names
* [ListenBrainz](https://listenbrainz.org) support for:
* Scrobbling and synchronizing listens
* Synchronizing 'love' feedbacks
* ReplayGain support
* User management, with several authentication backends, see [Deployment](INSTALL.md#deployment)
* User management, with several [authentication backends](INSTALL.md#authentication-backend)
* Subsonic API
## Music discovery
_LMS_ provides several ways to help you find the music you like:
* Tag-based filters (ex: _Rock_, _Metal_ and _Aggressive_, _Electronic_ and _Relaxed_, ...)
* Tag-based filters (ex: "_Rock_", "_Metal_ and _Aggressive_", "_Electronic_ and _Relaxed_", ...)
* Recommendations for similar artists and albums
* Radio mode, based on what is in the current playqueue
* Radio mode, with endless filling of the play queue with tracks similar to what is there
* Searches in album, artist and track names (including sort names)
* Starred Albums/Artists/Tracks
* Various tags to help you filter your music: _mood_, _albummood_, _albumgenre_, _albumgrouping_, ...
* Random/Starred/Most played/Recently played/Recently added for Artist/Albums/Tracks, allowing you to search for things like:
* Recently added _Electronic_ artists
* Random _Metal_ and _Aggressive_ albums
@@ -33,17 +33,8 @@ _LMS_ provides several ways to help you find the music you like:
* Starred _Jazz_ albums
* ...
The recommendation engine uses two different sources:
1. Tags that are present in the audio files
2. Acoustic similarities of the audio files, using a trained [Self-Organizing Map](https://en.wikipedia.org/wiki/Self-organizing_map)
__Notes on the self-organizing map__:
* training the map requires significant computation time on large collections (ex: half an hour for 40k tracks using a Core i5)
* audio acoustic data is pulled from [AcousticBrainz](https://acousticbrainz.org/). Therefore your audio files _must_ contain the [recording](https://musicbrainz.org/doc/Recording) [MusicBrainz Identifier](https://musicbrainz.org/doc/MusicBrainz_Identifier).
* to enable the audio similarity source, you have to enable it first in the administration panel.
## Subsonic API
The API version implemented is 1.16.0 and has been tested on _Android_ using _Subsonic Player_, _Ultrasonic_ and _DSub_.
The API version implemented is 1.16.0 and has been tested on _Android_ using _Subsonic Player_, _Ultrasonic_, _Symfonium_, and _DSub_.
Since _LMS_ uses metadata tags to organize music, a compatibility mode is used to browse the collection when using the directory browsing commands.
The Subsonic API is enabled by default.
@@ -53,9 +44,11 @@ __Note__: since _LMS_ may store hashed and salted passwords or may forward authe
_LMS_ relies exclusively on tags to organize your music collection.
### Filtering
You can specify the tags you want to be used to filter your collection. By default, `GENRE`, `ALBUMGROUPING`, `MOOD` and `ALBUMMOOD` tags are used.
You can specify the tags you want to use to filter your collection. By default, `genre`, `albumgrouping`, `mood` and `albummood` tags are used.
In the administration panel, you can set whatever tags you want, even custom tags.
__Note__: you can use the `lms-metadata` tool to have an idea of the tags parsed by _LMS_ using [TagLib](https://github.com/taglib/taglib).
### Multiple album artists
_LMS_ requires the `ALBUMARTISTS` and `ALBUMARTISTSSORT` tags to properly handle multiple album artists on the same album. As they are a custom tags, you may need to setup your favorite tagger to add them.
@@ -70,12 +63,6 @@ $setmulti(albumartistssort,%_albumartists_sort%)
* Previous track: <kbd>Ctrl</kbd> + <kbd>Left</kbd>
* Next track: <kbd>Ctrl</kbd> + <kbd>Right</kbd>
## Security considerations
_Wt_ (the web framework used) has some [built-in security measures](https://www.webtoolkit.eu/wt/features#security), but _LMS_ also has some too:
* to mitigate brute force login attempts, _LMS_ uses an internal login throttler based on the client IP address. The `Client-IP` or `X-Forwarded-For` headers are used to determine the real IP adress, so make sure to properly configure your reverse proxy to filter or even erase the values (see example in [INSTALL.md](INSTALL.md)).
* all passwords are stored hashed and salted using [bcrypt](https://fr.wikipedia.org/wiki/Bcrypt)
* all the resources relative to the music collection (tracks, covers, etc.) are private to an anthenticated session
## Installation
See [INSTALL.md](INSTALL.md) file.
approot/admin-database.xml
+2
View File
@@ -34,6 +34,7 @@
</div>
</div>
<legend>${tr:Lms.Admin.Database.scan-options}</legend>
${<if-has-recommendation-engine>}
<div class="col-lg-6">
<label class="form-label" for="${id:recommendation-engine-type}">
${tr:Lms.Admin.Database.recommendation-engine-type}
@@ -43,6 +44,7 @@
${recommendation-engine-type-info}
</div>
</div>
${</if-has-recommendation-engine>}
<div class="col-12">
<label class="form-label" for="${id:clusters}">
${tr:Lms.clusters}
src/libs/services/auth/impl/LoginThrottler.cpp
+23 -5
View File
@@ -54,7 +54,7 @@ LoginThrottler::removeOutdatedEntries()
for (auto it {std::begin(_attemptsInfo)}; it != std::end(_attemptsInfo); )
{
if (it->second <= now)
if (it->second.nextAttempt <= now)
it = _attemptsInfo.erase(it);
else
++it;
@@ -65,7 +65,6 @@ void
LoginThrottler::onBadClientAttempt(const boost::asio::ip::address& address)
{
const boost::asio::ip::address clientAddress {getAddressToThrottle(address)};
const Wt::WDateTime now {Wt::WDateTime::currentDateTime()};
if (_attemptsInfo.size() >= _maxEntries)
@@ -73,9 +72,25 @@ LoginThrottler::onBadClientAttempt(const boost::asio::ip::address& address)
if (_attemptsInfo.size() >= _maxEntries)
_attemptsInfo.erase(Random::pickRandom(_attemptsInfo));
_attemptsInfo[address] = now.addSecs(3);
AttemptInfo& attemptInfo {_attemptsInfo[address]};
if (attemptInfo.nextAttempt.isValid())
{
assert(attemptInfo.nextAttempt <= now); // should not be called if throttled
attemptInfo = {};
}
LMS_LOG(AUTH, DEBUG) << "Registering bad attempt for '" << clientAddress.to_string() << "'";
attemptInfo.badConsecutiveAttemptCount += 1;
LMS_LOG(AUTH, DEBUG) << "Registering bad attempt for '" << clientAddress.to_string() << "', consecutive bad attempts count = " << attemptInfo.badConsecutiveAttemptCount;
if (attemptInfo.badConsecutiveAttemptCount >= _maxBadConsecutiveAttemptCount)
{
LMS_LOG(AUTH, DEBUG) << "Throttling '" << clientAddress.to_string() << "'";
attemptInfo.nextAttempt = now.addMSecs(std::chrono::duration_cast<std::chrono::milliseconds>(_throttlingDuration).count());
}
else
{
attemptInfo.nextAttempt = {};
}
}
void
@@ -95,7 +110,10 @@ LoginThrottler::isClientThrottled(const boost::asio::ip::address& address) const
if (it == _attemptsInfo.end())
return false;
return it->second > Wt::WDateTime::currentDateTime();
if (!it->second.nextAttempt.isValid())
return false;
return it->second.nextAttempt > Wt::WDateTime::currentDateTime();
}
} // Auth
src/libs/services/auth/impl/LoginThrottler.hpp
+9 -3
View File
@@ -19,8 +19,7 @@
#pragma once
#include <shared_mutex>
#include <string>
#include <chrono>
#include <unordered_map>
#include <Wt/WDateTime.h>
@@ -44,8 +43,15 @@ namespace Auth
void removeOutdatedEntries();
const std::size_t _maxEntries;
static constexpr std::size_t _maxBadConsecutiveAttemptCount {5};
static constexpr std::chrono::seconds _throttlingDuration {3};
std::unordered_map<boost::asio::ip::address, Wt::WDateTime> _attemptsInfo;
struct AttemptInfo
{
Wt::WDateTime nextAttempt;
std::size_t badConsecutiveAttemptCount{};
};
std::unordered_map<boost::asio::ip::address, AttemptInfo> _attemptsInfo;
};
} // Auth
src/libs/services/scanner/CMakeLists.txt
-1
View File
@@ -1,6 +1,5 @@
add_library(lmsscanner SHARED
impl/AcousticBrainzUtils.cpp
impl/ScannerService.cpp
impl/ScannerStats.cpp
)
src/libs/services/scanner/impl/AcousticBrainzUtils.cpp
-87
View File
@@ -1,87 +0,0 @@
/*
* Copyright (C) 2018 Emeric Poupon
*
* This file is part of LMS.
*
* LMS is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* LMS is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with LMS. If not, see <http://www.gnu.org/licenses/>.
*/
#include "AcousticBrainzUtils.hpp"
#include <boost/property_tree/ptree.hpp>
#include <boost/property_tree/json_parser.hpp>
#include <Wt/WIOService.h>
#include <Wt/Http/Client.h>
#include "utils/IConfig.hpp"
#include "utils/Logger.hpp"
#include "utils/Service.hpp"
#include "utils/UUID.hpp"
namespace AcousticBrainz
{
static
std::string
getJsonData(const UUID& mbid)
{
static constexpr std::string_view defaultAPIURL {"https://acousticbrainz.org"};
const std::string url {std::string {Service<IConfig>::get()->getString("acousticbrainz-api-base-url", defaultAPIURL)} + "/api/v1/" + std::string {mbid.getAsString()} + "/low-level"};
boost::asio::io_service ioService;
Wt::Http::Client client {ioService};
client.setFollowRedirect(true);
client.setSslCertificateVerificationEnabled(true);
client.setMaximumResponseSize(256*1024);
if (!client.get(url))
{
LMS_LOG(DBUPDATER, ERROR) << "Cannot perform a GET request to url '" << url << "'";
return {};
}
std::string response;
client.done().connect([&](Wt::AsioWrapper::error_code ec, const Wt::Http::Message &msg)
{
if (ec)
{
LMS_LOG(DBUPDATER, ERROR) << "GET request to url '" << url << "' failed: " << ec.message();
return;
}
if (msg.status() != 200)
{
LMS_LOG(DBUPDATER, ERROR) << "GET request to url '" << url << "' failed: status = " << msg.status() << ", body = " << msg.body();
return;
}
response = msg.body();
});
ioService.run();
return response;
}
std::string
extractLowLevelFeatures(const UUID& recordingMBID)
{
return getJsonData(recordingMBID);
}
} // namespace Scanner::AcousticBrainz
src/libs/services/scanner/impl/AcousticBrainzUtils.hpp
-30
View File
@@ -1,30 +0,0 @@
/*
* Copyright (C) 2018 Emeric Poupon
*
* This file is part of LMS.
*
* LMS is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* LMS is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with LMS. If not, see <http://www.gnu.org/licenses/>.
*/
#pragma once
#include <string>
class UUID;
namespace AcousticBrainz
{
std::string extractLowLevelFeatures(const UUID& recordingMBID);
}
src/libs/services/scanner/impl/ScannerService.cpp
-81
View File
@@ -38,7 +38,6 @@
#include "utils/Logger.hpp"
#include "utils/Path.hpp"
#include "utils/UUID.hpp"
#include "AcousticBrainzUtils.hpp"
using namespace Database;
@@ -526,7 +525,6 @@ ScannerService::scan(bool forceScan)
if (!_abortScan)
{
checkDuplicatedAudioFiles(stats);
fetchTrackFeatures(stats);
reloadSimilarityEngine(stats);
}
@@ -560,85 +558,6 @@ ScannerService::scan(bool forceScan)
}
}
bool
ScannerService::fetchTrackFeatures(TrackId trackId, const UUID& recordingMBID)
{
std::map<std::string, double> features;
LMS_LOG(DBUPDATER, INFO) << "Fetching low level features for recording '" << recordingMBID.getAsString() << "'";
const std::string data {AcousticBrainz::extractLowLevelFeatures(recordingMBID)};
if (data.empty())
{
LMS_LOG(DBUPDATER, ERROR) << "Track " << trackId.getValue() << ", recording MBID = '" << recordingMBID.getAsString() << "': cannot extract features using AcousticBrainz";
return false;
}
{
auto uniqueTransaction {_dbSession.createUniqueTransaction()};
Track::pointer track {Track::find(_dbSession, trackId)};
if (!track)
return false;
_dbSession.create<TrackFeatures>(track, data);
}
return true;
}
void
ScannerService::fetchTrackFeatures(ScanStats& stats)
{
if (_recommendationServiceType != ScanSettings::RecommendationEngineType::Features)
return;
ScanStepStats stepStats{stats.startTime, ScanProgressStep::FetchingTrackFeatures};
LMS_LOG(DBUPDATER, INFO) << "Fetching missing track features...";
struct TrackInfo
{
TrackId id;
UUID recordingMBID;
};
const auto tracksToFetch {[&]()
{
std::vector<TrackInfo> res;
auto transaction {_dbSession.createSharedTransaction()};
auto trackIds {Track::findWithRecordingMBIDAndMissingFeatures(_dbSession, Range {})};
for (const TrackId trackId : trackIds.results)
{
const Track::pointer track {Track::find(_dbSession, trackId)};
res.emplace_back(TrackInfo {track->getId(), *track->getRecordingMBID()});
}
return res;
}()};
stepStats.totalElems = tracksToFetch.size();
notifyInProgress(stepStats);
LMS_LOG(DBUPDATER, INFO) << "Found " << tracksToFetch.size() << " track(s) to fetch!";
for (const TrackInfo& trackToFetch : tracksToFetch)
{
if (_abortScan)
return;
if (fetchTrackFeatures(trackToFetch.id, trackToFetch.recordingMBID))
stats.featuresFetched++;
stepStats.processedElems++;
notifyInProgressIfNeeded(stepStats);
}
notifyInProgress(stepStats);
LMS_LOG(DBUPDATER, INFO) << "Track features fetched!";
}
void
ScannerService::refreshScanSettings()
{
src/libs/services/scanner/impl/ScannerService.hpp
-2
View File
@@ -77,8 +77,6 @@ namespace Scanner
void scan(bool force);
void scanMediaDirectory( const std::filesystem::path& mediaDirectory, bool forceScan, ScanStats& stats);
bool fetchTrackFeatures(Database::TrackId trackId, const UUID& MBID);
void fetchTrackFeatures(ScanStats& stats);
// Helpers
void refreshScanSettings();
src/libs/subsonic/impl/SubsonicResource.cpp
+3 -3
View File
@@ -1883,8 +1883,8 @@ handleGetCoverArt(RequestContext& context, const Wt::Http::Request& /*request*/,
if (!trackId && !releaseId)
throw BadParameterGenericError {"id"};
std::size_t size {getParameterAs<std::size_t>(context.parameters, "size").value_or(256)};
size = Utils::clamp(size, std::size_t {32}, std::size_t {1024});
std::size_t size {getParameterAs<std::size_t>(context.parameters, "size").value_or(1024)};
size = Utils::clamp(size, std::size_t {32}, std::size_t {2048});
std::shared_ptr<Image::IEncodedImage> cover;
if (trackId)
@@ -2016,7 +2016,7 @@ static std::unordered_map<std::string, MediaRetrievalHandlerFunc> mediaRetrieval
// Media retrieval
{"/download", Stream::handleDownload},
{"/stream", Stream::handleStream},
{"/getCoverArt", handleGetCoverArt},
{"/getCoverArt", handleGetCoverArt},
};
src/lms/ui/LmsApplication.cpp
+1 -1
View File
@@ -401,7 +401,7 @@ LmsApplication::createHome()
R"(function(current) {
const menuItems = document.querySelectorAll('.nav-item a[href]:not([href=""])');
for (const menuItem of menuItems) {
if (menuItem.getAttribute("href").indexOf(current) !== -1) {
if (menuItem.getAttribute("href") === current) {
menuItem.classList.add('active');
}
else {
src/lms/ui/ModalManager.cpp
-2
View File
@@ -44,8 +44,6 @@ namespace UserInterface
void
ModalManager::show(std::unique_ptr<Wt::WWidget> modalWidget)
{
assert(count() == 0);
LMS_LOG(UI, DEBUG) << "Want to show, id = " << modalWidget->id();
std::ostringstream oss;
src/lms/ui/admin/DatabaseSettingsView.cpp
+10
View File
@@ -142,6 +142,13 @@ class DatabaseSettingsModel : public Wt::WFormModel
scanSettings.modify()->setClusterTypes(LmsApp->getDbSession(), std::set<std::string>(clusterTypes.begin(), clusterTypes.end()));
}
static ScanSettings::RecommendationEngineType getCurrentRecommendationEngine()
{
auto transaction {LmsApp->getDbSession().createSharedTransaction()};
const ScanSettings::pointer scanSettings {ScanSettings::get(LmsApp->getDbSession())};
return scanSettings->getRecommendationEngineType();
}
private:
void initializeModels()
{
@@ -211,6 +218,9 @@ DatabaseSettingsView::refreshView()
t->setFormWidget(DatabaseSettingsModel::UpdateStartTimeField, std::move(updateStartTime));
// recommendation engine type
// Hide the settings if the engine is set to clusters, as we don't want users to switch to acoustic features (currently broken)
// Otherwise, give the user a way to switch back to clusters
t->setCondition("if-has-recommendation-engine", model->getCurrentRecommendationEngine() == ScanSettings::RecommendationEngineType::Features);
auto recommendationEngineType {std::make_unique<Wt::WComboBox>()};
recommendationEngineType->setModel(model->recommendationEngineTypeModel());
t->setFormWidget(DatabaseSettingsModel::RecommendationEngineTypeField, std::move(recommendationEngineType));