DialogFlow CX + Google Chat - dealing with richContent

[french_williams]

I'm very new to DialogFlow - but I've figured out enough to set up a working DialogFlow + Google Chat integration based on a Cloud Storage bucket of unstructured data.  As it is, it's doing a very good job answering questions based on that data.

However, I'm struggling to figure out how to deal with the rich content that DialogFlow generates (citations, links, etc).  What I'm hoping to achieve is that the chatbot answers the questions and also provides links or buttons to the sources used for that answer (citations).

I'm pretty sure I have to process the JSON of the DialogFlow custom payload in a way that Google Chat can consume, but I have no idea how to do that.

Can someone point me in the right direction of how to handle this? 

Thanks!

[brandonwilliams]

same issues here - we were really excited about the chat integration and have a working model and have been really happy with the results of data store data we connected, but rich text/custom payloads we were never able to get to work. too, we struggled with truncation issues with dialogflow answers in chat. we are also very new to dialogflow, but def interested in following any advice here - thanks!

[xavidop]

Hi!

In order to do that you have to provide a custom a payload accepted by the Google Chat integration: https://cloud.google.com/dialogflow/es/docs/integrations/google-chat#custom-payload. To do that you just need to set up the custom payload on your page:

In that JSON you can use dialogflow variables.

Best,

Xavi

[JP_Contino]

Thats the snag, WHAT custom payload? Both as G products. I'd want to see some examples for all the typical use cases, to accelerate solution adoption.

[JP_Contino]

I would also be interested to know this, as it appears the Custom Payload referencing documentation is only showing-up as-expected in the default Dialogflow CX "Test Agent" web preview window. The moment, I cross over into GoogleChat or Slack integration, such Custom Payload is lost and no longer coming through.

FYI There are interesting references re Slack integration here https://github.com/GoogleCloudPlatform/dialogflow-integrations/blob/master/cx/slack/README.md which purport additional steps required to get it to work.
IMHO These manual steps/process needs refinement and documentation uplifted to make it clear that is the case.
We have spent quite a bit of time reading and re-reading documentation to make ends meet and make sense of this.

Judging by the comments, here, we're not alone. All further guidance and support very much welcomed.

[xavidop]

yeah I agree, the docs require more information.

Each integration supports its own custom payload. For example, I created a typescript repo to work with Dialogflow Messenger integration: https://github.com/xavidop/dialogflow-cx-messenger-ts
But for Slack for example, you can add these custom payloads: https://cloud.google.com/dialogflow/cx/docs/concept/integration/slack#rich_response_messages and you can find them here: https://api.slack.com/methods/chat.postMessage

Best,

Xavi

[JP_Contino]

We've spent much time on this, and happy to report back that we got it working but with much faff and workaround. Here is a much handy guide my collauge Zahid written to cover this topic of Dialogflow CX integration with Slack), albeit the process is likely thereabout similar. Hopefully it helps in your own journeys

https://medium.com/@zahid.saleem/decoding-google-clouds-genai-services-strengths-and-complexities-3f...

[xavidop]

this is such an amazing article! gonna share it with the Google Team and with the Google Community on Discord!

[french_williams]

Thanks for sharing this!  This is VERY helpful.  I appreciate it.

[JP_Contino]

super. 
we're exploring effective integration with GoogleChat (which iwll be the same behaviour) and then teams.

There is a certainly a good degree of value-add leaning on that `conversational AI` as a team resource in-chat, in company tools. which the data corpus can be updated with relevant content.

I mentioned this to Our Partner Eng, on last fridays GenAI Workshop at Google HQ, in London.

Just thinking of all the Partner Content one can discover 😅 with ease.

[xavidop]

yeah! hahaha it would be nice if we can do a talk or a workshop in person about this.

For dialogflow messenger integration (webchat) I would recommend to use my TS library: https://github.com/xavidop/dialogflow-cx-messenger-ts

Best,

Xavi

[Timur]

Hello,
I'm trying to use GoogleChat integration and I have similar problem with receiving a link to document in a response.
Could you please advise what I need to do in this Custom Payload?
Thank you in advance!

[xavidop]

can you please clarify which issues are you facing?

thanks

[Timur]

Hi 🙂

I added data sources with unstructured pdf files,
when I'm testing the bot via dialogflow messenger, I see the text response with information from pdf file and also clickable link to the document.
when I'm trying to use google chat integration the response from bot is only text, sometimes it's "Here is  a link that might be helpful" but there is no link.
I'm new to dialogflow so any information / advise on how to receive links in google chat integration will be helpful.
Thank you!

[xavidop]

Hi

you have to create a different response per channel. Please see the docs here: https://cloud.google.com/dialogflow/cx/docs/concept/fulfillment#channel-specific

That custom payload should include the structure that Google chat supports.

thanks.

[Timur]

Many thanks for you advise,
I added response per channel for google chat: 

but could you please also advise how to add a link for a document to Custom Payload json? 
as I understand fields could be taken from here: 

https://developers.google.com/workspace/chat/api/reference/rest/v1/spaces.messages#resource:-message

so I can use any of string fields, for example "formattedText": string, ?

and how to refer to a link, I tried "$richContent.actionLink" and
"text":"$request.knowledge.answers[0]",
"formattedText": "$richContent.citations.citation[0].actionLink"
but it didn't work.

[xavidop]

try this:

$sys.func.GET_FIELD($sys.func.GET($request.knowledge.sources[0], 0), "uri")

$sys.func.GET_FIELD($sys.func.GET($request.knowledge.sources[0], 0), "title")

[Timur]

thanks again for your response,

I tried both 

$sys.func.GET_FIELD($sys.func.GET($request.knowledge.sources[0], 0), "uri")

$sys.func.GET_FIELD($sys.func.GET($request.knowledge.sources[0], 0), "title")

but unfortunately I still don't receive a link in response in Google Chat

am I doing something wrong? I tried all options with and without quotes in $sys.func.GET_FIELD($sys.func.GET($request.knowledge.sources[0], 0), "uri") 😞

or the only way to receive a link in response is the way described by french_williams in this thread?

Thank you in advance!

[xavidop]

yeah,

My answer is more so if you want to get this from the flows and pages. If you want to get it from the webhook you have to follow what @french_williams did!

Best,

Xavi

[french_williams · Accepted solution]

So, after exploring a lot of options and with a lot of help from Google Gemini, I landed on a solution that worked for me.  One of my challenges was that I wanted to process the citations that were generated from my data store.   You have more flexibility with session parameters and using conditional payloads when using Routes+Intents vs data stores.  To work around that, I created 1 custom payload in my data store fulfillment that created hyperlinks to the 3 documents cited.  If there are less than 3 documents cited, it just returns blank lines.  Here's my entire process with some details (please don't judge my code - I'm not a developer 😉

Data Store Fulfillment setup

1. Agent says: $request.knowledge.answers[0] - to return the actual response from the query
2. Link maximum = 3
3. Custom payload
   
4. Enable a webhook that points to a Python Google Cloud function that processes the agent response and populates the session parameters for the 3 citation titles and 3 citation links.  Here's my Python Cloud Function Code (I probably don't need all of those imports - those were left over from some other testing):
   
5. Set the Transition to "End Session" so that it does not hang on to those session parameters in between inquiries.

The results look like this: