Skip to content

Commit

Permalink
feat: Support RSS3 UMS output format (DIYgod#13900)
Browse files Browse the repository at this point in the history 
* feat: support RSS3 UMS output

also optimized middleware/template

* fix: rss3_ums function name

* feat: handle current timestamp in UMS

* feat: add authors

* chore: update UMS type

* chore(deps-dev): bump @types/react from 18.2.37 to 18.2.38 in /website (#2249)

Bumps [@types/react](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/react) from 18.2.37 to 18.2.38.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/react)

---
updated-dependencies:
- dependency-name: "@types/react"
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps-dev): bump @types/markdown-it in /website (#2250)

Bumps [@types/markdown-it](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/markdown-it) from 13.0.6 to 13.0.7.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/markdown-it)

---
updated-dependencies:
- dependency-name: "@types/markdown-it"
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps-dev): bump @types/react-dom in /website (#2252)

Bumps [@types/react-dom](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/react-dom) from 18.2.15 to 18.2.16.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/react-dom)

---
updated-dependencies:
- dependency-name: "@types/react-dom"
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps-dev): bump @types/react-dom in /website (#2255)

Bumps [@types/react-dom](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/react-dom) from 18.2.16 to 18.2.17.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/react-dom)

---
updated-dependencies:
- dependency-name: "@types/react-dom"
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update lib/middleware/template.js

Co-authored-by: Tony <TonyRL@users.noreply.github.com>

* Update lib/middleware/template.js

Co-authored-by: Tony <TonyRL@users.noreply.github.com>

* Update lib/utils/render.js

Co-authored-by: Tony <TonyRL@users.noreply.github.com>

* Update lib/views/rss3-ums.js

Co-authored-by: Tony <TonyRL@users.noreply.github.com>

* Update lib/views/rss3-ums.js

Co-authored-by: Tony <TonyRL@users.noreply.github.com>

* Update lib/utils/render.js

Co-authored-by: Tony <TonyRL@users.noreply.github.com>

* doc: update UMS related docs

* Update parameter.md

Co-authored-by: Tony <TonyRL@users.noreply.github.com>

* Update parameter.md

Co-authored-by: Tony <TonyRL@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: pull[bot] <39814207+pull[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • Loading branch information
@HenryQW @pull @dependabot
3 people authored Nov 28, 2023
1 parent 82cf5c7 commit 3a5488a
Show file tree
Hide file tree
Showing 5 changed files with 147 additions and 87 deletions.
164 changes: 79 additions & 85 deletions lib/middleware/template.js
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
const { art, json } = require('@/utils/render');
const { art, json, rss3Ums } = require('@/utils/render');
const path = require('path');
const config = require('@/config').value;
const typeRegex = /\.(atom|rss|debug\.json|json|\d+\.debug\.html)$/;
const typeRegex = /\.(atom|rss|ums|debug\.json|json|\d+\.debug\.html)$/;
const { collapseWhitespace, convertDateToISO8601 } = require('@/utils/common-utils');

module.exports = async (ctx, next) => {
Expand All @@ -16,111 +16,105 @@ module.exports = async (ctx, next) => {

const outputType = ctx.state.type[1] || 'rss';

if (outputType === 'debug.json' && config.debugInfo) {
ctx.set({
'Content-Type': 'application/json; charset=UTF-8',
});
if (ctx.state.json) {
ctx.body = JSON.stringify(ctx.state.json, null, 4);
} else {
ctx.body = JSON.stringify({ message: 'plugin does not set debug json' });
}
}
if (outputType.endsWith('.debug.html') && config.debugInfo) {
ctx.set({
'Content-Type': 'text/html; charset=UTF-8',
});

const index = parseInt(outputType.match(/(\d+)\.debug\.html$/)[1]);
if (!(ctx.state.data && ctx.state.data.item && ctx.state.data.item[index])) {
ctx.body = `ctx.state.data.item[${index}] not found`;
} else {
ctx.body = ctx.state.data.item[index].description;
// only enable when debugInfo=true
if (config.debugInfo) {
if (outputType === 'debug.json') {
ctx.set({
'Content-Type': 'application/json; charset=UTF-8',
});
if (ctx.state.json) {
ctx.body = JSON.stringify(ctx.state.json, null, 4);
} else {
ctx.body = JSON.stringify({ message: 'plugin does not set debug json' });
}
}
}

if (outputType === 'json') {
ctx.set({ 'Content-Type': 'application/feed+json; charset=UTF-8' });
if (outputType.endsWith('.debug.html')) {
ctx.set({
'Content-Type': 'text/html; charset=UTF-8',
});

const index = parseInt(outputType.match(/(\d+)\.debug\.html$/)[1]);
if (!(ctx.state.data && ctx.state.data.item && ctx.state.data.item[index])) {
ctx.body = `ctx.state.data.item[${index}] not found`;
} else {
ctx.body = ctx.state.data.item[index].description;
}
}
}

if (!ctx.body) {
let template;

switch (outputType) {
case 'atom':
template = path.resolve(__dirname, '../views/atom.art');
break;
case 'rss':
template = path.resolve(__dirname, '../views/rss.art');
break;
default:
template = path.resolve(__dirname, '../views/rss.art');
break;
}
const templateName = outputType === 'atom' ? 'atom.art' : 'rss.art';
const template = path.resolve(__dirname, `../views/${templateName}`);

if (ctx.state.data) {
ctx.state.data.title = collapseWhitespace(ctx.state.data.title);
ctx.state.data.subtitle = collapseWhitespace(ctx.state.data.subtitle);
ctx.state.data.author = collapseWhitespace(ctx.state.data.author);

ctx.state.data.item &&
ctx.state.data.item.forEach((item) => {
if (item.title) {
item.title = collapseWhitespace(item.title);
// trim title length
for (let length = 0, i = 0; i < item.title.length; i++) {
length += Buffer.from(item.title[i]).length !== 1 ? 2 : 1;
if (length > config.titleLengthLimit) {
item.title = `${item.title.slice(0, i)}...`;
break;
}
}
const collapseWhitespaceForProperties = (properties, obj) => {
properties.forEach((prop) => {
if (obj[prop]) {
obj[prop] = collapseWhitespace(obj[prop]);
}

if (typeof item.author === 'string') {
item.author = collapseWhitespace(item.author);
} else if (typeof item.author === 'object' && item.author !== null) {
for (const a of item.author) {
a.name = collapseWhitespace(a.name);
}
if (outputType !== 'json') {
item.author = item.author.map((a) => a.name).join(', ');
});
};

collapseWhitespaceForProperties(['title', 'subtitle', 'author'], ctx.state.data);

ctx.state.data.item?.forEach((item) => {
if (item.title) {
item.title = collapseWhitespace(item.title);
// trim title length
for (let length = 0, i = 0; i < item.title.length; i++) {
length += Buffer.from(item.title[i]).length !== 1 ? 2 : 1;
if (length > config.titleLengthLimit) {
item.title = `${item.title.slice(0, i)}...`;
break;
}
}

if (item.itunes_duration && ((typeof item.itunes_duration === 'string' && item.itunes_duration.indexOf(':') === -1) || (typeof item.itunes_duration === 'number' && !isNaN(item.itunes_duration)))) {
item.itunes_duration = +item.itunes_duration;
item.itunes_duration =
Math.floor(item.itunes_duration / 3600) + ':' + (Math.floor((item.itunes_duration % 3600) / 60) / 100).toFixed(2).slice(-2) + ':' + (((item.itunes_duration % 3600) % 60) / 100).toFixed(2).slice(-2);
}

if (outputType !== 'rss') {
item.pubDate = convertDateToISO8601(item.pubDate);
item.updated = convertDateToISO8601(item.updated);
}

if (typeof item.author === 'string') {
item.author = collapseWhitespace(item.author);
} else if (typeof item.author === 'object' && item.author !== null) {
item.author.forEach((a) => (a.name = collapseWhitespace(a.name)));
if (outputType !== 'json') {
item.author = item.author.map((a) => a.name).join(', ');
}
});
}

if (item.itunes_duration && ((typeof item.itunes_duration === 'string' && item.itunes_duration.indexOf(':') === -1) || (typeof item.itunes_duration === 'number' && !isNaN(item.itunes_duration)))) {
item.itunes_duration = +item.itunes_duration;
item.itunes_duration =
Math.floor(item.itunes_duration / 3600) + ':' + (Math.floor((item.itunes_duration % 3600) / 60) / 100).toFixed(2).slice(-2) + ':' + (((item.itunes_duration % 3600) % 60) / 100).toFixed(2).slice(-2);
}

if (outputType !== 'rss') {
item.pubDate = convertDateToISO8601(item.pubDate);
item.updated = convertDateToISO8601(item.updated);
}
});
}

const routeTtl = (config.cache.routeExpire / 60) | 0;

const currentDate = new Date();
const data = {
lastBuildDate: new Date().toUTCString(),
updated: new Date().toISOString(),
ttl: routeTtl,
lastBuildDate: currentDate.toUTCString(),
updated: currentDate.toISOString(),
ttl: (config.cache.routeExpire / 60) | 0,
atomlink: ctx.request.href,
...ctx.state.data,
};

if (config.isPackage) {
ctx.body = data;
return;
}
if (!template) {
return;
}
if (outputType !== 'json') {

if (outputType === 'ums') {
ctx.set({ 'Content-Type': 'application/json; charset=UTF-8' });
ctx.body = rss3Ums(data);
} else if (outputType === 'json') {
ctx.set({ 'Content-Type': 'application/feed+json; charset=UTF-8' });
ctx.body = json(data);
} else {
ctx.body = art(template, data);
return;
}
ctx.body = json(data);
}
};
2 changes: 2 additions & 0 deletions lib/utils/render.js
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
const art = require('art-template');
const json = require('@/views/json');
const rss3Ums = require('@/views/rss3-ums');

// We may add more control over it later

module.exports = {
art,
json, // This should be used by RSSHub middleware only.
rss3Ums,
};
62 changes: 62 additions & 0 deletions lib/views/rss3-ums.js
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
const dayjs = require('dayjs');

/**
* This function should be used by RSSHub middleware only.
* @param {object} data ctx.state.data
* @returns `JSON.stringify`-ed [UMS Result](https://docs.rss3.io/docs/unified-metadata-schemas)
*/

const NETWORK = 'RSS';
const TAG = 'RSS';
const TYPE = 'feed';

const rss3Ums = (data) => {
const currentUnixTsp = dayjs().unix();
const umsResult = {
data: data.item.map((item) => {
const owner = getOwnershipFieldFromURL(item);
return {
owner,
id: item.link,
network: NETWORK,
from: owner,
to: owner,
tag: TAG,
type: TYPE,
direction: 'out',
feeValue: '0',
actions: [
{
tag: TAG,
type: TYPE,
platform: owner,
from: owner,
to: owner,
metadata: {
authors: typeof item.author === 'string' ? [{ name: item.author }] : item.author,
description: item.description,
pubDate: item.pubDate,
tags: typeof item.category === 'string' ? [item.category] : item.category,
title: item.title,
},
related_urls: [item.link],
},
],
timestamp: dayjs(item.updated).unix() || currentUnixTsp,
};
}),
};
return JSON.stringify(umsResult, null, 4);
};

// we treat the domain as the owner of the content
function getOwnershipFieldFromURL(item) {
try {
const urlObj = new URL(item.link);
return urlObj.hostname;
} catch (e) {
return item.link;
}
}

module.exports = rss3Ums;
3 changes: 2 additions & 1 deletion website/docs/parameter.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -136,14 +136,15 @@ There are more details in the [FAQ](/faq).

## Output Formats

RSSHub conforms to RSS 2.0, Atom and JSON Feed Standard, simply append `.rss`, `.atom` or `.json` to the end of the feed address to obtain the feed in corresponding format. The default output format is RSS 2.0.
RSSHub conforms to RSS 2.0, Atom, JSON Feed and RSS3 UMS Standard, simply append `.rss`, `.atom`, `.json`, or `.ums` to the end of the feed address to obtain the feed in corresponding format. The default output format is RSS 2.0.

E.g.

- Default (RSS 2.0) - [https://rsshub.app/dribbble/popular](https://rsshub.app/dribbble/popular)
- RSS 2.0 - [https://rsshub.app/dribbble/popular.rss](https://rsshub.app/dribbble/popular.rss)
- Atom - [https://rsshub.app/dribbble/popular.atom](https://rsshub.app/dribbble/popular.atom)
- JSON Feed - [https://rsshub.app/twitter/user/DIYgod.json](https://rsshub.app/twitter/user/DIYgod.json)
- RSS3 UMS - [https://rsshub.app/abc.ums](https://rsshub.app/abc.ums)
- Apply filters or URL query - [https://rsshub.app/dribbble/popular.atom?filterout=Blue|Yellow|Black](https://rsshub.app/dribbble/popular.atom?filterout=Blue|Yellow|Black)

### debug.json
Expand Down
3 changes: 2 additions & 1 deletion website/i18n/zh/docusaurus-plugin-content-docs/current/parameter.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -158,14 +158,15 @@ Telegram 即时预览模式需要在官网制作页面处理模板，请前往[

## 输出格式

RSSHub 同时支持 RSS 2.0、Atom 和 JSON Feed 输出格式，在路由末尾添加 `.rss`、`.atom` 或 `.json` 即可请求对应输出格式，缺省为 RSS 2.0
RSSHub 同时支持 RSS 2.0、Atom、JSON Feed 和 RSS3 UMS 输出格式，在路由末尾添加 `.rss`、`.atom`、`.json` 或 `.ums` 即可请求对应输出格式，缺省为 RSS 2.0

举例:

- 缺省 RSS 2.0 - [https://rsshub.app/jianshu/home](https://rsshub.app/jianshu/home)
- RSS 2.0 - [https://rsshub.app/jianshu/home.rss](https://rsshub.app/jianshu/home.rss)
- Atom - [https://rsshub.app/jianshu/home.atom](https://rsshub.app/jianshu/home.atom)
- JSON Feed - [https://rsshub.app/twitter/user/DIYgod.json](https://rsshub.app/twitter/user/DIYgod.json)
- RSS3 UMS - [https://rsshub.app/abc.ums](https://rsshub.app/abc.ums)
- 和 filter 或其他 URL query 一起使用 - `https://rsshub.app/bilibili/user/coin/2267573.atom?filter=微小微|赤九玖|暴走大事件`

### debug.json
Expand Down

0 comments on commit 3a5488a

Please sign in to comment.